Why out-of-order Stripe & Shopify webhooks resurrect canceled subscriptions — and how to build a 72h monotonic state machine

Verifying Stripe-Signature, X-Shopify-Hmac-Sha256, or X-RevClamp-Signature over the raw UTF-8 request body proves that an incoming webhook was genuinely emitted by the provider within your 300-second transport window. What cryptographic signature verification cannot guarantee is that two valid webhooks emitted 50 seconds apart will arrive at your application servers in chronological order.

In distributed billing pipelines across Stripe (2025-03-31.basil), Shopify Plus B2B, and Lemon Squeezy, webhook delivery queues provide at-least-once delivery, never in-order or exactly-once delivery. If your webhook handler blindly updates your database row using whatever valid event arrives last, a single worker stall or HTTP 502 retry on an earlier customer.subscription.updated (status: "active") event will arrive after customer.subscription.deleted (status: "canceled") and silently resurrect a canceled account into active paid access.

1. Anatomy of the “Zombie Subscription” race condition

Consider what happens when a B2B customer updates a billing seat or payment method at t = 1760000950 and then immediately cancels their subscription at t = 1760001000:

• Event A (evt_update_active_0950, created: 1760000950): Emitted first with status: "active" and tier: "growth_59", but experiences a 4-second TLS handshake timeout on Worker #1 and enters the provider's retry queue.
• Event B (evt_cancel_final_1000, created: 1760001000): Emitted 50 seconds later with status: "canceled" and tier: "free". It lands on Worker #2 immediately, which updates the tenant row to canceled.
• Delayed Retry of Event A (arrives at t = 1760001025): The provider retries Event A with a freshly signed HTTP transport timestamp header (passing your 300-second HMAC check!). A naive webhook handler executes UPDATE tenants SET status = 'active', tier = 'growth_59'—resurrecting the canceled subscription (RESURRECTED_CANCELED_SUBSCRIPTION) and leaking unpaid API or SaaS access indefinitely.

2. Five webhook ordering & retry failure modes across B2B billing engines

To prevent both duplicate side-effects and out-of-order state regressions, every stateful webhook consumer must separate HTTP transport freshness (the header signature timestamp) from domain state monotonicity (the event's logical creation watermark):

3. Atomic SQLite WAL / PostgreSQL CAS watermark pattern & live simulation

In production, your database update should enforce the monotonic watermark atomically inside a single compare-and-swap (CAS) statement so concurrent webhook workers cannot race between a SELECT and an UPDATE:

// Atomic monotonic watermark guard (only advances when incoming createdSec >= stored transition_seq)
await db.tenants.updateMany({
  where: {
    stripeCustomerId: payload.customer_id,
    transitionSeq: { lte: payload.created_sec }
  },
  data: {
    billingStatus: payload.subscription_status,
    tier: payload.tier,
    transitionSeq: payload.created_sec
  }
});

You can test any 1–10 event arrival sequence against both our 72-hour deduplication Set and monotonic lastAppliedCreatedSec watermark without an API key via our interactive browser simulator (/tools/webhook-replay-simulator), our 8th Model Context Protocol tool (revclamp_simulate_webhook_replay on POST https://api.revclamp.com/api/mcp), or a single CLI command:

# 1. One-shot CLI simulation of the out-of-order cancel-then-stale-update trap
npx -y github:lhphong09t2/revclamp-mcp --sim-replay out_of_order_stale_update_after_cancel

# 2. JSON-RPC 2.0 MCP tools/call payload (POST https://api.revclamp.com/api/mcp)
{
  "jsonrpc": "2.0",
  "id": "webhook-fsm-sim-1",
  "method": "tools/call",
  "params": {
    "name": "revclamp_simulate_webhook_replay",
    "arguments": {
      "scenarioPreset": "out_of_order_stale_update_after_cancel",
      "idempotencyWindowSec": 259200
    }
  }
}