Retries Are Normal: Building an Idempotent Sync API

In short: Networks retry, so any API that moves data or money receives some requests twice. I built an idempotent sync API and hit three instructive bugs: a JSON mismatch that failed without any error, deduplication that worked in tests but not over HTTP, and the design change that made retries free. The pattern applies to payments, webhooks and message queues alike.

When my laptop sends its changes to the VPS, the request can fail in the most annoying way possible: the VPS receives it, applies it, and the response gets lost. From the laptop's side that looks identical to "never arrived", so it sends the batch again.

Any network protocol that retries delivers messages at least once. The receiver has to make the second delivery harmless. That property is idempotency, and getting it right took three attempts.

Attempt 1: batch IDs

Each sync batch carries an ID. The receiver remembers IDs it has processed. If one arrives again, it answers "accepted, nothing applied" instead of applying it twice:

{"accepted": true, "applied": 0, "skipped": 1, "message": "Duplicate batch ignored"}

A duplicate is accepted, not rejected. The sender did nothing wrong; it just can't tell whether its first attempt landed. Returning an error would make it retry forever.

Unit tests passed. Then I tried it over HTTP.

The bug that returned no error

The sender encoded batches like this:

{"from_node": "laptop", "batch_id": "b-1", "mutations": [...]}

The receiver's Go struct had no JSON tags, so Go's decoder looked for keys named after the fields: FromNode, BatchID, Mutations. Nothing matched.

Here's the part that cost me time: json.Decode doesn't treat unknown keys as an error. It returned a perfectly valid, completely empty struct. The handler then correctly rejected an empty batch as invalid. The handler did its job on garbage input; the bug was that the encoder and decoder had two different ideas of what a batch looked like.

The fix was to put the tags on the one real type and have both sides use it:

type SyncBatch struct {
	FromNode  string         `json:"from_node"`
	BatchID   string         `json:"batch_id"`
	Mutations []SyncMutation `json:"mutations"`
}

One contract, used by both ends, can't drift apart. Two "equivalent" definitions eventually will.

Attempt 2: deduplication that only worked in tests

With the wire format fixed, I sent the same batch twice to a running server and wrote down my prediction first: second response, "Duplicate batch ignored". Actual result:

Accepted Applied Skipped Message
-------- ------- ------- -------
    True       1       0 Batch applied successfully
    True       1       0 Batch applied successfully

Applied twice. The cause was one line in the HTTP handler:

service := replication.NewSyncService() // inside the handler

A value created inside a request handler lives for one request. Every request got a fresh service with an empty "seen batches" set, so no batch was ever a duplicate. The unit tests reused one service across calls, so they never saw it.

The fix: create one service per node when the program starts, and share it with every request (with a mutex, since requests run concurrently). The replay then returned Applied: 0, Skipped: 1.

Attempt 3: IDs that make retries free

Where does a batch ID come from? A random ID works for network retries, but I wanted something stronger: if the laptop sends the same contents again (say, after a restart), the VPS should skip it without looking at a single change.

So the ID is a hash of the sender plus its contents:

func BatchIDFor(fromNode string, mutations []SyncMutation) string {
	hashes := make([]string, 0, len(mutations))
	for _, mutation := range mutations {
		hashes = append(hashes, HashMutation(mutation))
	}
	sort.Strings(hashes) // order-independent
	// ...sha256 of fromNode + hashes, first 8 bytes as hex
}

Could two different batches share an ID? Only if a machine's state returned to an earlier version, and vector clocks only move forward, so it can't.

Two layers, and why the second one matters

Deduplication now happens twice:

1. Batch ID seen before? Skip the whole batch. Fast.
2. Otherwise, per change: compare vector clocks. A change the receiver already has compares Equal or older and is skipped.

The second layer let me fix a slow memory leak safely. The set of seen batch IDs grew forever, so I capped it at 10,000 and forget the oldest. Is forgetting safe? Only because of layer 2: a forgotten batch that arrives again is processed, and every change in it is skipped anyway. I wrote a test that forgets a batch, replays it, and checks that nothing changes.

A test checks one replay. Later I wrote a TLA+ specification of the protocol and stated the property directly: delivering any message a second time, in any state any node can reach, changes nothing. The model checker verified it across every interleaving of three machines (see part 9). That's the real justification for forgetting batch IDs.

Where this shows up in real systems

• Payment APIs. Stripe's idempotency keys are the best-known example: resending a charge request with the same key can't charge the customer twice.
• Message queues. Kafka, SQS and most brokers deliver at least once by default. Consumers that aren't idempotent eventually process something twice.
• Webhooks. Providers retry when your endpoint times out, even if you'd already processed the event. Every webhook handler needs the "accepted, nothing applied" path.
• Contract drift between services. The silent JSON mismatch is the small version of a common integration failure: two teams' "equivalent" schemas that quietly disagree. Shared types or contract tests catch it; decoders that ignore unknown fields hide it.

What I took away

• Design for the second delivery from the start. A retry isn't an edge case.
• A decoder that ignores unknown fields can turn a schema mismatch into silent empty data. Keep one definition of every message.
• Test through the real transport. Unit tests with a long-lived object hid a bug that the first real HTTP request exposed.
• A fast path is fine to make lossy when there's a correct slow path behind it.

Next: How Two Computers Find Their Differences Without Sending Everything.

---

About this series: I'm learning distributed systems by building a real application, with an AI assistant as coach and pair programmer. It explains, reviews and sometimes writes code; I verify every claim against tests and measurements, and every bug and number here is real.

I write about building reliable software in Go, from the code up. If you're working on similar problems, I'd like to hear from you: get in touch or connect with me on LinkedIn.