An AI agent that starts work from a webhook is only as safe as its answer to one question: what happens when the same provider event shows up again?
Redelivery is normal. Timeouts, retries, and concurrent workers are normal. The failure mode is a second invoice, a second payout attempt, a second patient reminder, or a second MCP tool call that changes the same record. Idempotency is the design that keeps those repeats from becoming duplicate business effects.
This post is about event receive, durable enqueue, and once-only side effects when agents or automations act on provider webhooks. For choosing webhooks versus polling in the first place, see webhooks vs polling. For money-moving gates after an event is accepted, see finance AI agent approval gates. For binding an approval to one tool call across retries, see MCP permission retries and approval controls.
Providers warn you up front
Stripe's webhook documentation states that events may be delivered more than once and are not guaranteed to arrive in the order they were generated. It recommends logging processed event IDs, handling work asynchronously, and retrieving objects through the API when you need current state.
GitHub's webhook best practices call for HTTPS, secrets, unique delivery identifiers, and handling redelivery. A delivery ID is an operational handle for deduplication, not decoration.
Those vendor rules apply whether a human wrote the handler or an agent decided which tool to run after the event landed. The agent does not get a special exemption from redelivery physics.
Safe receive path before any agent step
Keep the HTTP edge boring and fast:
- 1Verify the signature or shared secret before trusting the payload.
- 2Parse a stable provider event ID (and delivery ID when the provider exposes one).
- 3Persist that ID in a durable store with a unique constraint, or look it up before enqueue.
- 4Put accepted work on a durable queue with the event ID as the correlation key.
- 5Return success quickly so the provider does not keep hammering a slow agent loop.
- 6Process the queue asynchronously: load context, decide tools, execute side effects.
If step 3 says the event was already accepted, return success again without enqueueing a second job. Rejecting the HTTP call with an error often makes the provider retry into a worse storm. Acknowledge the delivery; skip the duplicate work.
Do not run the LLM, call MCP tools, or touch money rails inside the request thread that must answer the provider within a few seconds. That path is how timeouts create "maybe it worked" ambiguity, which then becomes a blind retry.
Deduplicate before side effects
Store at least:
- provider name and event ID (unique together);
- delivery ID when available;
- received-at timestamp;
- processing state (accepted, in_progress, succeeded, failed_terminal);
- a hash or canonical form of the payload you acted on;
- outbound operation IDs produced by your side effects.
The unique key on provider + event ID is the guard that stops two workers from both treating the same Stripe `invoice.paid` (or equivalent) as new work. If a second delivery arrives while the first job is still `in_progress`, park it or no-op; do not start a parallel agent run that can race the first.
When the provider only gives a notification without a full object (common on some Google Workspace push channels), still key off the delivery or channel message identity they document, then fetch the current resource before mutating anything. The webhooks vs polling post covers that hybrid pattern in more detail.
Reconcile before retry
Timeouts and unknown outcomes need a different move than "run the agent again."
After a timeout, network cut, or worker crash:
- 1Look up the event ID and any outbound idempotency keys already issued.
- 2Ask the source system for current state (invoice status, appointment status, payment intent, ticket state).
- 3Ask your own ledger for whether the side effect already completed.
- 4Only then decide: no-op, continue a half-finished step, or surface an operator exception.
Blind retry is how you double-charge, double-email, or double-book. Stripe's API idempotent requests guidance exists for the outbound side of that problem: send a client-generated idempotency key with mutating calls so a retried HTTP request does not create a second object. Use the same idea for your own writes, even when the destination API is not Stripe.
Outbound side effects need their own once-only markers
Inbound event dedup stops a second *start*. Outbound idempotency stops a second *effect* when your worker retries after the first attempt may have succeeded.
Give every material side effect a stable key derived from the provider event ID plus the action name, for example `stripe_evt_123:send_receipt` or `gh_delivery_456:open_ticket`. Persist the key before the external call when you can, or rely on the destination's idempotency support when it has one.
Side effects that need this treatment in agent stacks:
- payment release, refund, or payout initiation;
- CRM create/update that must not fork duplicate records;
- email/SMS that would confuse a customer if sent twice;
- calendar write or EHR-adjacent update;
- MCP tool calls that mutate external systems;
- any write that an approval gate already authorized for one payload.
If a human approval was bound to one execute call, a redelivery must not open a second approval for a remapped tool. That is the same binding rule as MCP retry and remap controls: the first yes is not a blank check for a different call.
Agent-specific failure modes
Classic webhook handlers fail on redelivery and ordering. Agents add a few more:
| Failure mode | What goes wrong | Control |
|---|---|---|
| LLM or runner retries the whole turn | Same event triggers two tool sequences | Event-level lock + outbound idempotency keys |
| Concurrent workers dequeue the same job | Parallel mutations on one record | Unique event key + single-consumer lease |
| Tool remap after approval | Approved "release ACH" becomes a wider HTTP post | Bind approval to exact tool + args; reject remap |
| Reprocessing an old event from a dead-letter replay | Stale payload overwrites newer state | Reconcile current source state before write |
| "Helpful" agent re-reads the queue and re-acts | Duplicate customer-facing actions | Treat succeeded event IDs as terminal; require explicit replay with new audit |
| Long agent thinking past provider timeout | Provider retries while first run still live | Fast ack + durable queue; never do agent work in the webhook request |
Prompts that say "be careful not to duplicate" do not enforce any row in that table. Put the checks in the receive path, the queue lease, and the execute service.
A practical checklist before an agent may act on a webhook
Use this before you call the run production-ready:
| Check | Pass condition |
|---|---|
| Signature verify | Shared secret or asymmetric verify runs before parse/trust |
| Event identity | Stable provider event ID (and delivery ID if offered) is stored with a uniqueness rule |
| Ack timing | HTTP success returns after durable accept, before agent/tools |
| Duplicate delivery | Second delivery returns success and does not enqueue a second job |
| Queue lease | Only one worker holds `in_progress` for a given event ID |
| Reconcile path | Timeout/unknown outcome fetches source + local ledger before retry |
| Outbound keys | Mutating calls carry a stable idempotency key tied to event + action |
| Approval binding | Money or high-impact tools require named approval bound to exact payload |
| Operator visibility | Exhausted failures land in a queue a human can see, with event ID and last error |
| Replay policy | Manual replay is explicit, audited, and still goes through reconcile |
If any row is "we'll notice in logs," the design is not done. Logs help after the duplicate invoice exists. Uniqueness constraints and idempotency keys prevent it.
How this fits OrchestriAI delivery
On integration and agent builds, systems integration owns webhook receivers, durable queues, reconciliation jobs, and exception handling. AI agent systems put the run and human gates after the event is safely accepted. Custom MCP server development keeps mutating tools narrow and checks authorization on every call so a retried turn cannot widen into a different write.
The design goal is small and inspectable: each provider event is accepted once, processed under a lease, reconciled when outcome is unknown, and allowed to produce each outbound effect at most once. That is how AI agents survive webhook redelivery without teaching your customers that "the bot did it twice."
