Webhook Pipeline for Multi-Tenant Commerce Sync

The most interesting part of this one is not the architecture, it is that the problem was invisible until I went looking. The platform ingested orders from more than ten external commerce systems into our OMS, and the path was designed for eventual consistency: an order would eventually sync, so a few minutes of lag was considered fine. It was not fine. On a revenue-critical path, a customer placing an order waited up to roughly five minutes for it to move from "placed" to "processing" and verify on the OMS. Support had normalized it, the complaints were logged as "operational" and handled case by case, so the systemic delay had no owner. The hard part of this project was recognizing that an accepted status quo was quietly costing user experience and revenue, and that it did not have to be that way.

The platforms either already supported webhooks or could. So on the revenue-critical path I made the case to the OMS team to put webhooks on their roadmap, and when the slot came I picked it up and delivered. A shallow fix would have been to poll more often; I did not, because it would have raised cost without changing the shape of the problem.

I don't bet everything on a happy path, so before shipping webhooks I studied how they fail, and the worst case is that a webhook simply never fires. I did not rip polling out. I kept it as a slow safety net, dropped from every five minutes to once an hour. That hourly sweep re-syncs, and because idempotency is enforced at the destination, the OMS, with a deterministic token derived from each order's own identity, replaying an event converges to a single write rather than double-processing (a per-delivery UUID only dedupes the delivery, not a crash mid-pipeline between our database and the OMS), so any miss or drift self-heals at the next reconcile. Webhooks for speed, hourly polling for safety; the system never assumes the webhook arrived.

For the rest of the usual webhook pain, duplicates, out-of-order delivery, one noisy tenant, I built per-platform adapters that normalize each provider's format, idempotent dedup, per-tenant ordering where sequence mattered, dead-letter handling with replay so failures had a recoverable lane instead of silent loss, and per-tenant throttling so one platform could not drown the others. Observability mattered more than usual: once a system behaves close to real time, "it should catch up" stops being a satisfying answer, so the pipeline had to show where an event was, what failed, and what recovery path existed.

The outcome: freshness from 4+ minutes to sub-10 seconds, a user now sees "placed → processing" in about ten seconds, and Lambda spend down roughly 15% as the polling waste disappeared. The trade-off was accepting more reliability engineering in exchange for fresher truth: polling is easier to reason about because you own the schedule, and webhooks give some of that up, so you need the defensive patterns to go with the choice. I think of this less as "I moved to webhooks" and more as "I changed the contract", faster truth over timing ownership, and the first move was not code at all. It was noticing that everyone had quietly agreed to live with a problem.