One purchase, end to end

It's three seconds past ten on Friday, and Maya, in a London café, taps "buy 14C" on her phone. Half a second later her screen says "you got 14C."

That sounds instant. But in that half second her request crossed an ocean, hopped between tixdrop's services, and waited on a company tixdrop doesn't own. This lesson follows that one purchase, order 8812 for $40 with idempotency key k-7f3a from API design, and asks two questions: where did the half second go, and which of those milliseconds could you remove, move out of the way, or simply stop waiting for?

Across the Ocean

Maya's phone is in London and tixdrop runs in Virginia. Her request takes about 40 ms to get there and 40 ms to come back: 80 ms, and no code you write in Virginia makes a signal cross the Atlantic faster.

That is the good case, because her phone already had a connection open. On a brand new connection, three more things happen first:

A cold connection costs 260 ms of pure travel before tixdrop does a single millisecond of work. Hold on to that number; it gets cut in half later.

Hop by Hop

Now follow the request inside Virginia, one stop at a time. This is the grown-up tixdrop from the start of the course, plus one box: a risk check that scores each purchase for fraud.

Each hop inside Virginia also costs about 0.5 ms of network round trip (the number from service discovery). That is too small to matter here, so the budget leaves it out.

Every arrow in this purchase carries the whole module so far:

• a TCP connection (TCP), usually one already open in a pool,
• encrypted with TLS, and inside Virginia both ends show certificates, which is mutual TLS (TLS),
• pointed at an address found by service discovery (service discovery),
• and on the charge, Maya's idempotency key k-7f3a, so a retry can't charge her twice (API design).

Draw each hop as a bar on one time axis and you get a staircase called a waterfall:

                          0         100       200       300       400       500 ms
                          |---------|---------|---------|---------|---------|---
phone to Virginia      40 ████
load balancer           1     ▏
accounts                3     ▏
seat hold (primary)     8     █
risk check            120      ████████████
payments + processor  300                  ██████████████████████████████
commit order 8812       8                                                █
enqueue email           2                                                 ▏
Virginia to phone      40                                                 ████

That last small bar, the email, is the next section's story.

The Critical Path

Maya also needs a confirmation email with her seat and order number. The obvious version: orders calls the mail server before it answers her. The mail server takes 700 ms, so the purchase takes 1.22 seconds, and for most of it Maya watches a spinner while a mail server finishes a job she doesn't care about yet.

Before tixdrop tells her yes, it needs the seat hold, the charge and the order. It does not need the email. The chain of steps a person has to wait for is the critical path, and the email was never on it; the naive design put it there.

So orders drops a message on the queue from the original tixdrop diagram, which takes 2 ms, and a worker sends the email a moment later. 1.22 seconds becomes 0.52. That one move saves more than everything else in this lesson combined.

The general method: for any request you design, list the steps the caller really waits for, and move everything else off the path. Steps that don't depend on each other can also run side by side, so the critical path is the longest chain of dependent steps, not the sum of all the work.

Taking work off the path doesn't make it free. The waiting and the failures move somewhere you still have to handle them. The mail server can be down, so the worker needs retries; orders can crash between committing order 8812 and enqueuing the email, so the message can be lost unless the commit and the enqueue are tied together (the outbox pattern in transactions); and a queue delivers at least once, so the worker must not send two emails for one order (messaging).

What Must Stay on the Path

If moving the email helped that much, could the charge come off the path too? Plenty of shops do exactly that: they show "order received, payment pending" and charge the card a minute later.

tixdrop doesn't, and the reason is what Maya does with the answer. The moment she sees "you got 14C," she closes the app and texts her friends that she's going. If her card is declined a minute later, she's already gone, and tixdrop has promised her a seat it now has to take back. The hold on 14C also can't wait forever for her to come back with another card while other fans are trying to buy it.

So for tickets, the charge stays on the path. The rule:

• Go asynchronous when the caller doesn't need the result to act (the email, analytics, a receipt PDF).
• Stay synchronous when she does (the charge that decides whether 14C is hers).

An online store that ships in two days can reasonably charge asynchronously, because nothing happens before the charge settles. It still needs an honest "payment pending" state and a plan for the declined card.

Someone Else's 300 ms

Look at the biggest bar left. Of the 440 ms inside Virginia, 300 belong to the card processor, and the processor isn't tixdrop's. You can't rewrite its code, add servers to it, or move it closer to your data center.

The one thing you do control is how long you are willing to wait for it, and what you tell Maya when that time runs out. On a bad day this bar stretches far past 300 ms, and the answer needs to be ready before that day.

Moving the Handshake Closer

Back to the cold connection: 260 ms of handshakes and travel before any work. The handshakes are slow because every round trip crosses the Atlantic. So end them in London instead.

tixdrop already serves its poster from a content delivery network. Let its London edge take Maya's connection too. DNS still costs 20 ms, but the TCP and TLS handshakes now happen 10 ms away. The edge keeps warm connections open to Virginia, so her request rides one that is already set up: 10 ms to the edge, 80 ms across the ocean and back on the warm connection.

Half the cost, just by moving where the handshakes happen. On a warm connection the edge doesn't help (it adds a 10 ms detour), because the request still has to cross the ocean to reach the seats. The edge is a fix for new connections; the other fixes are not opening new connections in the first place (keep-alive, HTTP/2 reuse) and making the ones you do open cheaper (TLS session resumption from the TLS lesson).

One in a Hundred, Five Times

So far every bar has had one fixed length. Real services wobble. Inside Virginia, orders makes five calls in a row: accounts, the seat hold, the risk check, payments, and the commit. Say each usually answers quickly, but one time in a hundred it takes over a second, because of a garbage collection pause, a busy disk, or a queue behind someone else's request.

One in a hundred sounds rare enough to ignore. Out of every hundred purchases like Maya's, how many hit at least one slow call?

Each call is fine 99 times in 100, so all five are fine only 0.99⁵ ≈ 95.1% of the time:

P(at least one slow call) = 1 - 0.99^5 ≈ 4.9%   (about 1 purchase in 20)

Out of 5,000 buyers on Friday, about 245 stare at a spinner for more than a second. Every piece was slow one time in a hundred, and the purchase as a whole is slow one time in twenty. Those rare slow responses are the latency tail, usually described by high percentiles: p99 is the latency that 99% of requests come in at or under. When calls are chained, the tail of the whole is worse than the tail of any part.

Fan-out makes it dramatic. Jeff Dean and Luiz André Barroso worked through this at Google in "The Tail at Scale" (2013), with servers that usually answer in 10 ms and take a full second one time in a hundred:

If a user request must collect responses from 100 such servers in parallel, then 63% of user requests will take more than one second.

(1 - 0.99¹⁰⁰ ≈ 63%.) Every call you add, in a row or side by side, is another roll of the dice. The same paper describes the main countermeasure, hedged requests: if a response hasn't arrived by the time most responses would have, send a second copy to another replica and use whichever answers first. That is only safe for reads or for requests carrying an idempotency key, and the timeouts and retries lesson covers when it pays.

Deadlines

Give the processor its bad day: it is struggling, and Maya's charge will take 4 seconds.

Orders promised her phone an answer within 1.5 seconds; after that, the phone shows "confirming your payment." But payments doesn't know about that promise, and its own rule says to wait up to 10 seconds for the processor. So long after orders gives up, payments still holds a connection and a worker, waiting on a charge nobody is waiting for. Multiply that by every slow purchase on Friday, and payments runs out of workers doing work no one will ever read.

The fix is to send the time along with the request:

Payments stops waiting at about 1.3 s and answers orders in time. That remaining time, passed from hop to hop, is a deadline.

Send it as a remaining duration, not a wall-clock time, because the two machines' clocks don't agree exactly (the clocks lesson explains why). The gRPC documentation:

To address this gRPC converts the deadline to a timeout from which the already elapsed time is already deducted.

In Go with gRPC, setting it once at the edge of orders is enough; every call made with that context carries what is left:

// orders: the whole purchase gets 1.5 s
ctx, cancel := context.WithTimeout(ctx, 1500*time.Millisecond)
defer cancel()
 
if err := accounts.Check(ctx, session); err != nil { ... }
if err := seats.Hold(ctx, "14C", 8812); err != nil { ... }
if err := risk.Score(ctx, order); err != nil { ... }
 
// payments receives the remaining time (about 1.37 s) as grpc-timeout,
// and its call to the processor inherits it through ctx
charge, err := payments.Charge(ctx, &ChargeRequest{
    Order: 8812, AmountCents: 4000, IdempotencyKey: "k-7f3a",
})

Over plain HTTP there is no standard deadline header, so teams pass one of their own and every service has to read it, subtract its own time, and forward it. Either way, a server should also check the deadline before starting expensive work: if it has already passed, the answer is useless. Timeouts, retries and their budgets get their own full treatment in the timeouts and retries lesson.

Unknown, Again

Look closely at what just happened to Maya's charge. Payments stopped waiting at about 1.3 seconds, and the processor may have charged her card a moment later, or never. From where payments sits, the outcome is unknown, the same bin from the API design lesson.

So her phone keeps showing "confirming your payment" and asks again, with the same key, k-7f3a. Payments passes the key to the processor, which recognizes it. If the first charge went through, it hands back that same charge; if it didn't, it makes it now. Either way Maya pays $40 exactly once.

The deadline and the idempotency key are two halves of one design: the deadline lets every hop give up on time, and the key makes giving up safe.

Reading a Trace

The waterfall you have been reading is real, and you can get one for every purchase:

1. Give the purchase one trace id at the front door.
2. Pass that id along with every call.
3. Have each service record when its step started and ended.
4. Line those records up by time.

The result is a distributed trace, and each bar is a span. The id travels in a standard header, traceparent, defined by W3C Trace Context:

traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01

The four fields are the format version, the 16-byte trace id shared by every span in the purchase, the id of the span that made this call (the parent), and flags such as whether the trace is sampled. In the spec's words:

The traceparent header represents the incoming request in a tracing system in a common format, understood by all vendors.

To read one, start with the longest bar, then look for gaps between bars (time spent waiting in a queue or a pool that no service recorded), then check whether the steps really had to run in order. The observability lesson shows how to collect traces from a whole fleet.

The Map

The start of the course drew tixdrop's grown-up shape. Every box on it gets a lesson of its own later:

The Takeaway

That is the whole communication module on one purchase. Every arrow is TCP, encrypted with TLS, and pointed at an address from service discovery. The charge carries the idempotency key that makes a retry safe. This lesson put a price on each arrow, in milliseconds and in chances to fail.

The one sentence to keep: every arrow costs time and adds a place to fail, and the budget tells you which ones are worth your attention.

Every arrow also ends the same way: one machine waiting on another, deciding from silence alone whether it is slow or gone. That is where the next module begins. When the primary database suddenly goes quiet, is it dead, or should you wait? That's system models, next.

References

• The Tail at Scale (Dean and Barroso, Communications of the ACM, February 2013)
• Deadlines | gRPC
• Trace Context (W3C Recommendation)