The Refund That Ran Twice
A payment processor fires a webhook to confirm a refund. Your automation catches it, processes the refund, and updates the customer record. Clean. Except the payment processor also retried that webhook three seconds later — because your server responded a hair too slow, or a network hiccup swallowed the 200 response. Now the customer has two refunds. The money is already out the door.
This is not a hypothetical edge case. It is the default behavior of nearly every webhook-based integration. Stripe, PayPal, Square, Shopify — every major payment platform documents that webhooks may be delivered more than once. They tell you this plainly. And yet the automations built on top of those webhooks routinely ignore it.
The fix is a concept with an ungainly name and an elegant principle: idempotency. Processing an event once and processing it five times should produce exactly the same result. Getting this right is foundational work — the soil preparation that determines whether your automation bears fruit or quietly bleeds cash.
Why Duplicate Webhooks Are the Rule, Not the Exception
Understanding why duplicates happen matters, because it shapes how you design against them. Webhook delivery operates on an at-least-once model. The sender fires the event and waits for an acknowledgment. If the acknowledgment does not arrive — due to a timeout, a dropped connection, a slow response, or a temporary server error — the sender retries. Most platforms retry multiple times with exponential backoff, sometimes over hours or even days.
This means your system will receive duplicates whenever:
- Your endpoint takes longer than the sender's timeout threshold to respond (often as short as 5 seconds).
- A load balancer, proxy, or CDN between you and the sender drops or delays the response.
- Your server returns a 200 but the sender's network does not receive it.
- You deploy or restart your application during the delivery window.
- The sender itself has a bug or an internal retry storm.
None of these require your code to be broken. They require your code to be prepared.
Idempotency: The Principle Behind Exactly-Once Processing
An idempotent operation produces the same outcome whether it runs once or many times. In the context of webhook-driven automations, this means: no matter how many times a given event arrives, the side effects — the refund, the inventory adjustment, the email, the ledger entry — happen exactly once.
Note the precision here. The goal is not to prevent duplicate delivery (you cannot control that). The goal is to make duplicate delivery harmless. This is sometimes called exactly-once processing, though the more accurate description is at-least-once delivery with idempotent consumption.
The Idempotency Key
Most payment platforms include a unique event identifier in every webhook payload — an event ID, a delivery ID, or both. This identifier is the same across retries. It is your idempotency key.
The core pattern is straightforward:
- Receive the event. Extract the unique event identifier from the payload.
- Check your record. Look up that identifier in a persistent store (a database table, a key-value store, even a simple ledger).
- If already processed, stop. Return a 200 to acknowledge receipt, but do not execute the side effect again.
- If new, process and record. Execute the business logic, then write the event identifier to your store — ideally within the same transaction.
That is the skeleton. The engineering is in the details.
Where the Pattern Gets Hard: Five Real Trade-Offs
1. The Race Condition Window
If two identical webhook deliveries arrive within milliseconds of each other — and both hit your lookup before either has written the record — both will see the event as new and both will process it. This is not theoretical. Under load, with horizontally scaled workers, it happens.
The fix is an atomic check-and-set operation. In a relational database, this means using a unique constraint on the event identifier column and letting the database reject the duplicate insert. The second worker's INSERT fails, and you handle that failure gracefully by returning a 200 without processing. In Redis, a SETNX (set-if-not-exists) with a TTL achieves the same atomicity. The key insight: your data store is the arbiter, not your application logic.
2. Choosing the Right Identifier
Not all identifiers in a webhook payload are equally useful. Some platforms send both an event ID (unique to the event itself) and a delivery ID (unique to each delivery attempt). You want the event ID, because it stays the same across retries. Using the delivery ID would defeat the purpose — every retry looks new.
Read your payment processor's documentation carefully. If no stable event ID exists, you may need to construct a composite key from the event type, the resource ID (e.g., the payment or order ID), and a timestamp or version number. This is less clean but still effective.
3. Transaction Boundaries
The most dangerous failure mode is processing the side effect (issuing the refund) but failing to record the event identifier (database write times out, application crashes between the two steps). On retry, the event looks new again, and the refund fires a second time.
The most grounded solution: wrap the side effect and the record-write in a single database transaction where possible. If the side effect is an external API call — like actually issuing a refund via a payment API — you cannot include it in a database transaction. In that case, the pattern shifts:
- Record the event as in-progress (claim it).
- Execute the external API call.
- Update the record to completed on success, or failed on error.
Retries that see an in-progress record can either wait or skip, depending on your tolerance for delay versus duplication. This is where engineering judgment matters — the right choice depends on what the automation is doing and what happens if it does it twice.
4. How Long to Keep the Deduplication Record
Event identifiers need to persist long enough to cover the sender's retry window. If a payment processor retries over 72 hours, your deduplication store needs to remember events for at least that long. After that, a TTL-based expiration is reasonable for most cases.
For financial operations, consider keeping the record permanently. Storage is cheap. A double refund is not.
5. Responding Before Processing
A tempting shortcut: return the 200 immediately, then process the event asynchronously. This reduces the chance of timeouts and retries — but it introduces its own complexity. If the async processing fails, you have already acknowledged the event. The sender will not retry. You need your own retry mechanism, your own dead-letter queue, your own monitoring.
This is a valid architecture for high-volume systems, but it trades one class of problems for another. For automations that touch money, the simpler synchronous path — with idempotency built in — is often the more grounded choice.
Beyond Webhooks: Idempotency Across Your Automation Stack
The principle extends well beyond payment webhooks. Any workflow automation that triggers side effects from external events needs this thinking:
- CRM-to-email automations. A contact update event fires twice. Does the customer receive two welcome sequences?
- Inventory sync workflows. A stock adjustment event replays. Does inventory drop by double the actual quantity?
- Multi-agent orchestration systems. An AI agent receives a duplicate task dispatch. Does it execute the task twice, doubling the cost or the output?
Anywhere an event can arrive more than once — and in distributed systems, that is everywhere — idempotent design is not a nice-to-have. It is the foundation.
A Practical Checklist for Idempotent Webhook Processing
Before any webhook-driven automation goes live, walk through this:
- Have you identified the stable, unique event identifier in the payload?
- Are you storing processed event identifiers in a persistent, durable store?
- Is the check-and-set atomic — will a race condition between two simultaneous deliveries still result in only one execution?
- Is the side effect and the deduplication write as close to a single transaction as possible?
- If the side effect is an external API call, do you have an in-progress state that prevents re-execution?
- Is your deduplication store retention at least as long as the sender's maximum retry window?
- Are you returning a 200 for successfully deduplicated events (not a 4xx, which would trigger more retries)?
- Do you have monitoring or alerting for duplicate events, so you can see how often they occur?
This checklist is not exhaustive, but it catches the failures that actually cost money.
Why This Matters More as Automations Scale
A single webhook integration with one payment processor is manageable. But as workflow automation grows — connecting payment events to accounting systems, to customer notifications, to inventory, to reporting pipelines — the surface area for duplicate processing multiplies. Each connection point is another place where at-least-once delivery can create real damage if the consumer is not idempotent.
This is especially true in systems that use AI-driven workflow automation, where an LLM or multi-agent system is making decisions based on incoming events. If the event stream feeding those decisions contains undetected duplicates, the downstream effects compound. An AI agent that processes a refund event twice might also trigger two customer communications, two ledger adjustments, and two support tickets. The automation scales — and so does the error.
Building idempotency into the foundation is the kind of soil work that does not look dramatic but determines whether the system bears reliable fruit at scale.
The Deeper Point
Reliability in automation is not about preventing every possible failure. It is about designing systems that behave correctly when failures happen — because in distributed systems, they will. Idempotency is one of the most grounded, practical expressions of that principle. It accepts the reality of duplicate delivery and engineers around it, rather than hoping it will not occur.
At FigTree Development, we architect automation workflows with this kind of production rigor because we build on the same principles ourselves. Our own internal pipelines — from content production to publishing and distribution — are designed with idempotent processing, human approval steps, and the monitoring to catch what the automation misses. It is the same discipline we bring to every engagement: foundational work that compounds over time.
Related service: AI automation and workflow services in Tampa Bay — what we build, how we work, and what it costs.
See Where Your Automations Are Exposed
If you are building workflow automations that touch payments, inventory, or any process where a duplicate event means real cost, the architecture decisions you make now will determine how much you trust the system later. We help teams identify the highest-leverage points to engineer reliability — the places where getting idempotency right eliminates an entire class of problems.
Book a free 20-minute discovery call with us at FigTree Development. We will map where your automations are most exposed to duplicate processing and what it would take to make them grounded and scalable from day one.