Enterprise Webhook Dead-Letter Queue (DLQ), 72h Idempotency & Out-of-Order Recovery Runbook
When an upstream cloud load balancer returns 502 Bad Gateway or 504 Gateway Timeout during a deployment or traffic spike, Stripe (2025-03-31.basil), Shopify (2025-04), and Lemon Squeezy queue retries on divergent backoff schedules. Replaying failed webhooks blindly into a naive handler can resurrect canceled subscriptions, double-credit wholesale Net-30 deposits, or miss Shopify's 4-hour subscription auto-deletion cutoff. This runbook documents exact provider SLAs, our 4 deterministic recovery invariants, and an interactive DLQ replay command generator. Updated .
1. Retry Schedule & Timeout SLAs: Stripe vs. Shopify vs. Lemon Squeezy
Each billing and commerce platform enforces a distinct HTTP response deadline and retry curve. Understanding these boundaries prevents silent webhook subscription deletion and out-of-order state corruption during incident recovery.
| Provider & API Version | Strict HTTP Timeout SLA | Automatic Retry Schedule & Backoff Window | Terminal Exhaustion Behavior & DLQ Action |
|---|---|---|---|
Stripe (2025-03-31.basil) |
10s–20s TLS response deadline before marking attempt timed out |
72 hours (259,200s) exponential backoff across up to ~15 delivery attempts |
Events older than 72h stop retrying; replay via stripe events resend <evt_id> or customer subscription snapshot sync |
Shopify Plus B2B (2025-04) |
Strict 5s timeout (must acknowledge HTTP 200 OK in < 5,000ms) |
8 retries over 4 hours with exponential backoff across webhook topics | Auto-deletes webhook subscription after 4h of continuous failure; requires Admin GraphQL webhookSubscriptionUpdate + Net-Terms reconciliation |
| Lemon Squeezy (MoR) | 10s HTTP response timeout on POST /api/webhooks/lemonsqueezy |
3 retries at +5m, +25m, and +125m (4 total attempts over ~2.6 hours) |
Marked failed after +125m attempt; replay via Dashboard Settings -> Webhooks -> Resend or POST /api/webhooks/lemonsqueezy |
RevClamp Outbound (x-revclamp-signature) |
10s bounded egress timeout with 300s timestamp skew tolerance |
Deterministic idempotent transition_seq watermark on revoke / restore |
Downstream merchant middleware applies WHERE transitionSeq < payload.transition_seq atomic update |
| EU VIES SOAP/REST Gateway | 10s per lookup with circuit-breaker hold on MS_UNAVAILABLE |
Bounded to MAX_VIES_RETRIES = 3 (4 total attempts) before queued hold |
Places B2B order on safe PENDING_ASYNC_RETRY fulfillment hold without charging 0% VAT prematurely |
2. The 4 Recovery Invariants Enforced During DLQ Replay
Because RevClamp Enterprise enforces all four invariants below at the storage engine layer, on-call engineers can safely replay entire dead-letter queues or overlapping time windows without risking duplicate side effects or resurrected canceled entitlements.
| Recovery Invariant | Storage & State-Machine Field | Exact Enforcement Rule | Anomaly Prevented |
|---|---|---|---|
| 1. 72-Hour Event ID Deduplication | seenBillingEvents (SQLite seen_billing_events table, 259,200s TTL) |
If seenBillingEvents.has(eventId) within the 259,200s (72h) window, return 200 OK with DUPLICATE_EVENT_IDEMPOTENT_NOOP. Events older than 72h return EXPIRED_OUTSIDE_72H_DEDUP_WINDOW. |
Prevents duplicate retry storms (502/504 timeouts after DB commit) from double-crediting invoices or re-triggering downstream webhooks. |
| 2. Monotonic Timestamp Watermark | latestBillingEventCreatedSec / lastAppliedCreatedSec |
Requires event.createdSec >= latestBillingEventCreatedSec (and createdSec <= nowSec + 34560000). Stale arrivals return STALE_OUT_OF_ORDER_TRANSITION_IGNORED. |
Prevents a delayed customer.subscription.updated (active) webhook from overwriting a newer customer.subscription.deleted (canceled) event. |
| 3. Terminal-Event Precedence | subscription_cancelled, customer.subscription.deleted, order_refunded, expired |
At equal timestamps (createdSec === latestBillingEventCreatedSec), terminal cancellation/refund states cannot be overwritten by non-terminal updates, and expired/unpaid is evaluated before cancel_at_period_end (ends_at). |
Prevents same-second race conditions or elapsed ends_at timestamps from granting unpaid grace access. |
| 4. SQLite WAL Atomic Persistence | BEGIN IMMEDIATE ... COMMIT (journal_mode = WAL, busy_timeout = 5000) |
Persists seen_billing_events and updates tenant latestBillingEventCreatedSec + tier in one atomic SQLite WAL transaction with 90-day rolling TTL pruning. |
Eliminates split-brain state between in-memory caches and disk across K3s pod restarts or concurrent webhook deliveries. |
3. Interactive DLQ Recovery Command & Pre-Flight Replay Generator
Select your webhook provider, incident failure mode, and workspace identifier below to generate the exact 3-step recovery runbook: (a) Pre-Flight Dry-Run Simulation (via MCP CLI or REST), (b) Provider-Specific Safe Replay Step, and (c) Post-Replay Verification Check (GET /api/dashboard/billing-status).
out_of_order_stale_update_after_cancel (259,200s window) so stale updates with createdSec < latestBillingEventCreatedSec are safely ignored before executing stripe events resend.
# ============================================================================
# REVCLAMP 3-STEP WEBHOOK DLQ RECOVERY PLAN
# Provider: Stripe (2025-03-31.basil) | Mode: Out-of-Order Stale Delivery
# Workspace: tnt_enterprise_prod_01 | Event ID: evt_basil_dlq_recover_01
# ============================================================================
# STEP 1 (PRE-FLIGHT DRY-RUN): Verify 72h dedup & monotonic watermark behavior
npx -y github:revclamp/revclamp-mcp --sim-replay out_of_order_stale_update_after_cancel 259200
curl -sS -X POST https://api.revclamp.com/api/tools/webhook-replay-simulate \
-H "Content-Type: application/json" \
-d '{"scenarioPreset":"out_of_order_stale_update_after_cancel","idempotencyWindowSec":259200}'
# STEP 2 (SAFE PROVIDER REPLAY): Resend event via Stripe CLI (2025-03-31.basil)
# Monotonic guard drops stale createdSec < latestBillingEventCreatedSec automatically
stripe events resend evt_basil_dlq_recover_01 \
--webhook-endpoint we_revclamp_prod \
--live
# STEP 3 (POST-REPLAY VERIFICATION): Confirm workspace watermark & entitlement state
curl -sS https://revclamp.com/api/dashboard/billing-status \
-H "X-RevClamp-Tenant-Key: <YOUR_TENANT_KEY>"
Related Engineering Guides, Diagnostic Sandboxes & Independent Disclosure
FTC & Independent Architecture Disclosure: RevClamp Enterprise is an independent B2B revenue-assurance and tax-compliance software platform (0% dispute commission forever, no paid affiliate partnership or compensated affiliate links). Stripe, Shopify, and Lemon Squeezy are trademarks of their respective owners. All retry windows, GraphQL mutations, and CLI recovery sequences on this page are deterministic engineering runbooks verified against production webhook specifications.