API design: the $40 retry

Maya is on café wifi in London, and she has just tapped "buy" on seat 14C for $40. Her phone sends the purchase to tixdrop's orders service in Virginia, and orders asks the payments service to charge her card. Payments passes that on to the card processor, an outside company, which approves the charge in about 300 ms. Payments records it, and its reply to orders is lost on the way.

Orders has been waiting for 2 seconds, and its timer runs out.

This is exactly where the first lesson left off. So... just retry it? If orders does, Maya might pay $80 for one seat. If it doesn't, she might pay $40 for no seat at all.

This lesson has two halves. First, make that retry completely safe. Second, change this API later without breaking the phones already out there.

Done, Not Done, Unknown

Before fixing anything, sort what orders can actually observe when a call to payments fails. Not every failure is the same kind of mystery.

The line between the bins is whether the request left orders and reached code that could act on it. Not done is safe: retry it (a refused connection can go to another instance) or fix it. Done is done. Every call in unknown might have charged Maya's card, and nothing orders saw can tell you which.

During the drop about one call in 200 times out, so of 5,000 purchases, 25 end up in unknown. That bin is the whole problem.

Never Retry, or Always Retry?

There are two simple policies for those 25, and both are wrong.

Never retry. Orders tells Maya something went wrong. But her card may already be charged, so some of those fans pay $40 and walk away with no seat, and the rest lose a seat they would have gotten.

Always retry. The theory is that the first charge probably didn't happen. Orders sends the charge again, payments has no idea it is a repeat, and the processor does exactly what it is told. Maya's card statement gets a second line: $40, again. Across the drop, that is up to 25 fans charged twice for one seat.

Both policies guess. What you need is a charge that is safe to send twice.

Safe to Repeat by Design

Some operations are already safe to repeat. Look at the seat hold, where orders writes to the seat map, "seat 14C is held by order 8812." Send it once and the seat is held. Send it three times and the seat is still held, by the same order, exactly once. It sets a state, so every repeat lands in the same place. In HTTP that is a PUT.

The charge, "charge $40 to this card," adds something. Three copies are $120. In HTTP that is a POST.

The repeats don't need the same response. The second PUT might say "already held" instead of "held," and that's fine, because what happened on the server is the same. RFC 9110, the HTTP spec, defines it by effect:

A request method is considered "idempotent" if the intended effect on the server of multiple identical requests with that method is the same as the effect for a single such request.

The method is a promise, not an enforcement mechanism: a PUT handler that appends a row is not idempotent because of its verb. When you can, design operations as "set to" rather than "add," and a lot of retry problems disappear.

A sibling tool solves a different problem. If-Match makes an update conditional on the version you last read, so two writers can't silently overwrite each other:

PUT /seats/14C/hold
If-Match: "v41"
Content-Type: application/json
 
{ "order": 8812 }

If the seat changed since version 41, the server answers 412 Precondition Failed. That guards against lost updates between different requests. Duplicates of the same request need something else, because the charge is not idempotent, so you have to make it that way.

The Idempotency Key

Before orders sends the charge, it writes a label on it: a short random key, k-7f3a, that means "this one purchase." Every retry of Maya's charge carries the same label, and a different purchase gets a different one. Payments keeps a table with three columns: the key, its state, and the result it sent back.

For now, pretend only one request with a given key ever arrives at a time, and build the obvious version:

1. A charge arrives. Payments looks up k-7f3a and finds nothing.
2. It charges the card and saves k-7f3a | done | 200 charge ch_91.
3. The reply gets lost, and orders retries with the same key.
4. This time the lookup finds the row, so payments doesn't touch the processor. It sends back the saved answer, and Maya's statement still shows one line.

That label is an idempotency key. It turns a charge into something you can safely send twice.

The Race

Now drop the "one at a time" pretense, because retries don't wait politely for each other.

The processor is slow today, and Maya's first charge is still out there when orders' 2-second timer fires. The retry reaches payments while the first attempt is still waiting on the processor. Payments looks up k-7f3a and finds nothing, because the first attempt only saves its row at the end. So the retry charges the card too, and two lines appear on Maya's statement, with a perfectly good key table sitting right under them.

The same happens if two copies of the request land 50 ms apart: both look, both see an empty table, both charge. The gap between checking the table and acting on it is a race, and "check, then act" is its usual shape.

Claim the Key First

Close the gap by flipping the order: claim the key first, then do the work. When a charge arrives, payments inserts the row as a single step, and the database's uniqueness rule lets exactly one insert for that key succeed:

CREATE TABLE idempotency_keys (
  key          text PRIMARY KEY,
  request_hash text NOT NULL,          -- fingerprint of the request body
  state        text NOT NULL,          -- 'in_progress' or 'done'
  status_code  int,
  response     jsonb,
  created_at   timestamptz NOT NULL DEFAULT now()
);
 
-- the claim: exactly one request per key gets a row back
INSERT INTO idempotency_keys (key, request_hash, state)
VALUES ('k-7f3a', 'sha256:9c1e...', 'in_progress')
ON CONFLICT (key) DO NOTHING
RETURNING key;

Only the request that won the insert talks to the processor, and when the answer comes back it fills in the result. That single, all-or-nothing insert is what "claim the key atomically" means. Every other request with the same key reads the existing row and decides:

const won = await claim(key, hash)
if (!won) {
  const row = await getRow(key)
  if (row.request_hash !== hash) return reply(422) // same key, different request
  if (row.state === "in_progress") return reply(409) // still working, ask again
  return reply(row.status_code, row.response) // replay the saved answer
}
const result = await processor.charge({ amountCents: 4000, idempotencyKey: key })
await finish(key, result) // state = 'done', saved status and body
return reply(result.status, result.body)

What should payments answer the retry that arrives while the first attempt is still in progress? It can't say "success," because the first attempt might still be declined, and it certainly can't charge. The honest answer is "conflict, I'm still working on this one, ask me again in a moment." Stripe's API does exactly this with 409 Conflict. Orders waits a moment, asks again, gets the saved result, and Maya ends up with one charge and one line on her statement.

Charged, Then Crashed

There is one more gap, and it hides inside a single request. The processor charges Maya's card and sends back "approved," and then the payments server crashes before it can write the result. Her card has a $40 line, but the key table still says in_progress, and the process that owned the row is gone.

That is the crash window: the side effect happened somewhere else, and your record of it didn't. No table of yours can close it, because the charge lives at the processor, not in your database.

The processor can close it, if it is handed the same key. So payments passes k-7f3a along on its own call, and the processor keeps a key table of its own. When payments picks up the stale attempt (a row stuck in_progress longer than any charge takes) and sends the charge again with the same key, the processor answers "already approved, here's the original." The key travels hop to hop, from orders to payments to the processor, and every hop with a side effect checks it.

In 1984, Saltzer, Reed and Clark wrote this down in "End-to-End Arguments in System Design." Discussing a client that retries after a failure, they put it in one line: "the retried request is now a duplicate, which only the application can discover."

TCP can't see that duplicate; a retry on a new connection is a brand new byte stream as far as TCP is concerned. Only the code that knows what "one purchase" means can recognize it.

Two related cases. When the side effect lives in your own database, put the effect and the key row in the same transaction, and the window closes locally; the transactions lesson builds on that. When a downstream provider has no idempotency keys, send your own reference (order 8812) with the charge and, before retrying a stale attempt, ask the provider whether a charge with that reference exists; reconcile against its records daily either way.

Keep the Answer

Four rules keep keys honest. Stripe's documentation spells out the first two nicely.

Save the whole answer, errors included. If Maya's card was declined, the retry should hear "declined" again, not start a fresh attempt that might go through. Stripe:

Subsequent requests with the same key return the same result, including 500 errors.

Bind the key to the request it came with. Suppose a buggy client reuses k-7f3a for $400 instead of $40. Replaying the $40 answer to a $400 question would be wrong, and so would charging. Store a fingerprint of the request and reject a mismatch. Stripe: "The idempotency layer compares incoming parameters to those of the original request and errors if they're not the same to prevent accidental misuse."

Keep keys longer than any retry can take to show up. tixdrop keeps them for 24 hours. After that, the same key starts a brand new charge, so the retention window is part of the contract.

A key is a random label, never personal data. Not Maya's email, not her card number; keys end up in logs and tables all over the place. k-7f3a is shortened for reading. A real key is a random UUID (v4) or similar, with enough randomness that two clients never collide. Scope keys to the caller's account, too, so one client's key can never replay another client's answer.

Born at the Tap

So far orders has been making the key. But the purchase really starts when Maya taps "buy," so that is where k-7f3a should be born: minted by her phone at the tap.

On café wifi her phone might retry on its own, and she might tap twice because nothing seemed to happen. With the key minted at the tap, every one of those copies carries the same label, so orders, payments and the processor all see one purchase, all the way down. If orders minted a fresh key per request instead, those two taps would look like two perfectly valid purchases.

None of this works unless every caller knows the rules, so they go into the API contract:

POST /v1/orders
Idempotency-Key: k-7f3a
Content-Type: application/json
 
{ "event": "friday-drop", "seat": "14C" }

• Send an Idempotency-Key header with every purchase, minted once per intent.
• Reuse it, unchanged, on every retry of that purchase.
• Expect 409 while the first attempt is still running; wait and retry.
• Expect 422 if the same key arrives with a different body.
• Keys are remembered for 24 hours.

The Idempotency-Key header is a widely used convention (Stripe and many others) rather than a finished standard. An IETF draft that would standardize it, including the 409 and 422 answers above, has expired without becoming an RFC.

$40 Becomes $4,000

The second half of API design is changing an API without breaking anyone who uses it.

Here is the reply orders sends to Maya's phone:

{ "order": 8812, "seat": "14C", "price": 40.00 }

One day a team decides prices should be whole cents, to avoid rounding errors, so the same field now says:

{ "order": 8812, "seat": "14C", "price": 4000 }

The new app knows to divide by 100 and shows $40. But 12% of fans are still running last year's app, and it has no idea anything changed. If Maya's phone is one of them, it shows her a $4,000 ticket. Nothing crashed, no error fired, and every test the team wrote for the new app passed.

The field kept its name and changed its meaning. That is the most dangerous change you can make, because old clients can't tell. The rule: never redefine a field. Add a new one, keep the old one exactly as it was, and remove it only when nobody reads it:

{ "order": 8812, "seat": "14C", "price": 40.00, "price_cents": 4000 }

Two Versions at Once

Old clients aren't only phones in pockets. They are also your own servers, halfway through a deploy. When tixdrop rolls out a new orders service, old and new servers answer side by side for about ten minutes, so Maya's phone can get an old reply and then a new one a second later. Phones are slower still: some copies of last year's app stay installed for years.

Three habits cover almost every change:

1. Tolerant readers. Clients ignore fields they don't know, so last year's app skips price_cents instead of crashing. A client that validates responses strictly (rejecting unknown fields, or assuming an enum is closed) turns every additive change into a breaking one.
2. Expand, then contract. Add the new field next to the old one (expand). Move every reader over. Once nobody reads the old one, remove it (contract). Measure "nobody reads it" from real traffic, not from a guess.
3. Test across versions. Run old clients against the new server before the deploy, and new clients against the old server, because during a rolling deploy the new app can land on an old server that has never heard of price_cents.

A /v2 in the path is for the rare change you can't make by adding, and then both versions run until the old one's traffic reaches zero. Versioning through a header instead (Accept: application/vnd.tixdrop.v2+json) keeps URLs stable, but any cache in between must key on it, so the response needs Vary: Accept, and your logs need to record which version was served.

With Protocol Buffers the same idea shows up as field numbers: the number, not the name, is what goes on the wire. Never change an existing field's number or type, and never reuse a deleted one:

message OrderReply {
  int64  order       = 1;
  string seat        = 2;
  reserved 3;              // was: double price (removed after contract)
  reserved "price";
  int64  price_cents = 4;
}

Choosing a Wire Format

The same rules apply whatever carries the call. JSON over HTTP is readable, debuggable with curl, and understood by every client and proxy; it is the default for public APIs and phones. gRPC uses Protocol Buffers over HTTP/2, with generated clients, compact messages and streaming; it is common between services where you control both ends. An OpenAPI or .proto file makes the contract machine-readable, and a compatibility check in CI catches a removed field before a phone does. Neither format makes a charge idempotent or a field's meaning stable; that part is design.

The Takeaway

Walk Maya's purchase one last time. Her phone mints k-7f3a at the tap, every hop carries it, payments claims it before doing any work, and the processor checks it too, so a lost reply is just a question you can safely ask again. And the reply she gets back only ever gains fields; it never changes what one of them means.

The one sentence to keep: design every call for the unknown outcome, then evolve it by adding, never by redefining.

That one tap crossed an ocean and several services before Maya saw her ticket. Where did the time go, hop by hop, and which of those hops did she really have to wait for? That's one purchase, end to end, next.

References

• RFC 9110: HTTP Semantics (section 9.2.2, Idempotent Methods)
• Idempotent requests | Stripe API Reference
• End-to-End Arguments in System Design (Saltzer, Reed, Clark, 1984)
• The Idempotency-Key HTTP Header Field (IETF draft, expired)
• Language Guide (proto 3): Updating a Message Type