TCP: the first arrow

At the end of the last lesson, tixdrop had grown into a shape: a load balancer in Virginia, app servers behind it, a database, and arrows between all of them. This lesson zooms into one arrow, the first one a purchase crosses: from a fan's phone to that load balancer.

The fan is Maya. She is in a café in London, on café wifi, and tixdrop is 80 ms away there and back. At 10:00:03 she taps "buy 14C". If that request doesn't arrive whole, someone else gets her seat.

Her phone turns the tap into about 2,000 bytes: headers, her session cookie, the seat. The network carries data in packets, and each packet carries at most 1,460 bytes of data on a typical Ethernet path, so "buy 14C" is two packets. The 500 KB poster she looked at a moment earlier was 343.

Every one of those packets crosses the same wire, and as we saw at the start of the course, packets get lost, delayed, and reordered. So how does "buy 14C" arrive whole, and what does that promise cost in time? The answer is TCP, and the best way to understand it is to build it one piece at a time.

What the Wire Does

Send Maya's two packets across and watch what can happen to them.

A packet can vanish. Every router has a queue in front of each outgoing link. When packets arrive faster than the link can send them, the queue fills and the router drops whatever doesn't fit. Wifi adds its own losses on top. Nobody sends Maya's phone an error, and nobody warns tixdrop that something was on its way.

Packets can overtake each other. They sit in different queues, cross parallel links, or get handled by different queues inside a network card, and the second half lands first. (Routers normally keep one connection's packets on one path, so "each packet takes a different route" is not the usual cause.)

A packet can arrive twice. Usually because a copy was resent while the original was only running late.

So the load balancer might get the second half first, the first half twice, or only one half at all. The wire does all of this silently. Any fix has to come from the two ends, working it out between themselves.

Number Every Byte

The smallest thing the phone can add to each packet is a number. TCP's twist is that it numbers every byte, not every packet.

Lay Maya's request out on a ruler, 2,000 bytes starting at zero. The first packet says "I start at 0" and carries 1,460 bytes. The second says "I start at 1,460" and carries the remaining 540. That label is the sequence number: the position of the packet's first byte. Its length tells the receiver where it ends.

With that one field, the receiver can undo two of the wire's tricks:

• Reordering. If the packet starting at 1,460 arrives first, the load balancer knows where it goes and holds it until the gap before it fills. Your application only ever reads bytes in order.
• Duplication. If the packet starting at 0 arrives twice, the copy covers bytes the receiver already has, so it throws the copy away.

Why bytes and not packets? Because a resend doesn't have to reuse the original packet boundaries. The sender can cut the same bytes into bigger or smaller pieces, and the receiver still knows exactly where each one belongs.

Acknowledge What Arrived

Numbers sort things out, but a lost packet still leaves a hole the phone knows nothing about. So the receiver talks back.

That reply is an acknowledgment, or ACK. It always names the next byte the receiver wants, so "ACK 2,000" means bytes 0 through 1,999 all arrived. It says nothing about byte 2,000 itself. Because it is cumulative, one ACK confirms everything before it, and a lost ACK is covered by the next one. Receivers also delay ACKs slightly so one ACK can cover two packets.

One at a Time Is Too Slow

Now the obvious design: send one packet, wait for its ACK, then send the next. If no ACK comes back, send it again. This is called stop-and-wait, and it is perfectly reliable.

For "buy 14C" it is fine: two packets, two round trips, 160 ms.

Now turn the arrow around and send Maya the poster. 343 packets, each waiting a full 80 ms round trip for its ACK:

343 packets x 80 ms = 27.4 seconds

tixdrop's 1 Gbit/s link could push the same 500 KB out in 4 ms. With stop-and-wait, the wire sits empty 99.98% of the time. Reliable, and you would never ship it.

Many in Flight: The Window

The fix is to stop waiting. The sender puts a whole batch of packets on the wire back to back, and the ACKs stream home while more data is already on its way.

The sender allows a set number of bytes to be in flight: sent but not yet acknowledged. That allowance is the window. Picture it on the ruler: everything left of the window is acknowledged, everything inside it may be sent, everything to the right has to wait. Each ACK that comes back slides the window forward. That is a sliding window.

How big does the window need to be? The wire stays full only if the sender can keep sending for a whole round trip before the first ACK gets back. So the window has to hold one round trip's worth of the link:

bandwidth x round trip = 1,000,000,000 bits/s x 0.080 s
                       = 80,000,000 bits
                       = 10 MB in flight

That product is the bandwidth-delay product, and it is the first number to check when a transfer is slow on a fast link. Run it backwards and you get the ceiling any window puts on one connection:

The first row is history. The original TCP header had a 16-bit window field, so no window could exceed 65,535 bytes. At 80 ms that caps a single connection at about 6.5 Mbit/s whatever the link. Window scaling (RFC 7323) fixes it: both ends agree during setup to shift the window field left by up to 14 bits, which allows windows up to about 1 GB. Every modern stack uses it.

Scaling removes the header limit, not every limit. The receiver's socket buffer caps the window too (on Linux, net.ipv4.tcp_rmem), and so do middleboxes that strip the scaling option. When a fast link carries a slow transfer to somewhere far away, check the window before you blame the network.

When a Packet Is Lost

Now lose one packet in the middle of the poster, the one starting at byte 14,600. The packets behind it keep arriving, and each one makes Maya's phone repeat the same ACK.

Fast retransmit. Three duplicate ACKs are strong evidence that one packet is missing while later ones are getting through, so the sender resends the missing packet right away (RFC 5681). Why three and not one? Because a single duplicate is often just reordering, and the missing packet shows up a moment later.

Selective acknowledgment (SACK). A plain cumulative ACK can only say "I still need 14,600". With SACK, the ACK also lists the ranges the phone already holds past the hole, here 16,060 up to 20,440, so the sender resends only the hole instead of everything after it. Both ends negotiate SACK during setup, and practically every stack supports it.

The retransmission timeout. Fast retransmit needs packets behind the loss. If the very last packet is lost, nothing behind it triggers a duplicate ACK and the sender hears only silence. So it runs a timer. RFC 6298 sets it from a smoothed round-trip estimate plus four times its variation, starting at 1 second before any measurement exists. When the timer fires, the sender resends the oldest unacknowledged data and doubles the timer for next time. That doubling, exponential backoff, is the first backoff in this course; the timeouts and retries lesson comes back to it for application retries.

Here is the uncomfortable part. When the timer fires, the sender can't tell "my data was lost" from "the ACK coming back was lost". The silence looks identical. It is the lost reply from the first lesson all over again, one layer down. TCP's answer is to resend anyway, and let the sequence numbers make it harmless: if the data had arrived, the receiver discards the copy.

The Handshake

One thing so far was simplified: the ruler started at zero. A real connection starts its numbering at an unpredictable point. RFC 9293 derives it from a clock plus a keyed hash of the two addresses and ports, which makes it hard for a leftover packet from an old connection, or a stranger guessing, to land on numbers that look valid.

The cost is that each side has to announce its starting number before any data flows. That is the three-way handshake:

1. SYN (synchronize). The phone sends its starting sequence number, x.
2. SYN-ACK. The load balancer sends its own starting number, y, and acknowledges x + 1. The SYN itself uses up one sequence number, which is why the ACK is x + 1.
3. ACK. The phone acknowledges y + 1. This packet can already carry the first bytes of "buy 14C".

The SYN goes out and the SYN-ACK comes back: one full round trip, 80 ms, before the first byte of "buy 14C" leaves Maya's phone. On a new connection her request costs two round trips (handshake, then request and response), 160 ms. On a connection that is already open, it costs one.

The handshake sets up transport state and nothing more. It doesn't prove who is on the other end, and it doesn't mean any application behind the load balancer is ready. TCP Fast Open can put data in the SYN on repeat connections, but it is rarely usable on the public internet: middleboxes interfere, and data in a SYN can be delivered twice.

Flow Control

So far everything protects the data from the wire. The receiver can be overwhelmed too.

Each connection on the load balancer has a receive buffer, where bytes wait until the program behind it reads them. If that program falls behind, the buffer fills, and anything more would have nowhere to go.

Every ACK carries a number saying how much room is left: the receive window. The sender never has more unacknowledged bytes out than that. When the buffer is full the window drops to zero and Maya's phone stops sending until room opens up. While the window is zero, the sender sends small probes now and then, so a lost "room again" update can't stall the connection forever.

That is flow control: the reader sets the pace. A slow reader never loses data; the writer just slows down. The window limits bytes outstanding, not bytes per second.

Congestion Control

Flow control protects the receiver. Nothing built so far protects the network in between.

Picture a router in the middle of Maya's path, shared by thousands of connections, with a queue in front of its link. When packets arrive faster than the link can send them, the queue fills and the router drops what doesn't fit. Every sender sees its losses and does exactly what it was built to do: resend. The resends land in the same full queue, so more are dropped and more are resent. Soon the link is busy all the time and almost nothing useful gets through.

This isn't a thought experiment. Van Jacobson and Michael Karels open their 1988 paper with it:

In October of '86, the Internet had the first of what became a series of 'congestion collapses'. During this period, the data throughput from LBL to UC Berkeley (sites separated by 400 yards and two IMP hops) dropped from 32 Kbps to 40 bps.

Eight hundred times slower, across a distance you could walk in five minutes.

Their fix is still in every TCP: a second window, kept by the sender for the network, called the congestion window (cwnd). The sender grows it while ACKs come back smoothly and cuts it when a loss says a queue somewhere is full. The amount actually in flight is limited by both windows:

in flight <= min(congestion window, receive window)

So when a queue overflows, every sender through it backs off, and the queue drains.

How much to grow and how hard to cut is the congestion control algorithm, and there are three you will meet:

• Reno (RFC 5681) is the classic. After the startup phase it grows the window by about one packet per round trip. Three duplicate ACKs halve it; a timeout drops it to one packet and starts over. Plotted over time it is the sawtooth in every textbook.
• CUBIC is the default on Linux, Windows and macOS. It cuts less on loss (to 70%) and grows back along a cubic curve based on time since the last loss, so it refills fast, long paths much quicker than Reno.
• BBR, from Google, doesn't treat loss as the main signal. It estimates the path's bottleneck bandwidth and minimum round trip, then paces packets to match, which keeps router queues short.

Routers that support ECN (RFC 3168) can mark a packet "I'm congested" instead of dropping it, which gives senders the signal without the loss.

Slow Start and Short Transfers

How big should the congestion window be on a brand new connection, when the sender knows nothing about the path?

It starts small and learns. Most servers start at 10 packets, 14.6 KB (RFC 6928 raised the initial window to that in 2013), and double it every round trip that completes without loss. That ramp is called slow start; it is only slow next to sending the whole window at once.

Load the poster on a cold connection. For now, ignore server time and assume nothing is lost during the ramp.

Seven round trips, 560 ms, for an image the link could push out in 4 ms. A short transfer never reaches the link's speed; it finishes while the window is still growing. So for small responses, and most API responses are small, you are not waiting on bandwidth at all. You are waiting on round trips. A faster link does nothing for Maya; a closer server or fewer round trips does.

The rule of thumb for a cold transfer of N packets with an initial window of 10:

round trips = 1 (handshake) + smallest r where 10 x (2^r - 1) >= N
poster:       1 + 6 = 7 round trips, 7 x 80 ms = 560 ms

Change the round trip or the loss rate and three stories fall out:

• 10 ms round trip (a fan near Virginia): the whole load takes 70 ms and nobody notices the handshake.
• 200 ms round trip: the handshake alone is a fifth of a second, and the poster takes 1.4 s.
• 1 packet in 50 lost (a bad day on café wifi): even a warm connection keeps cutting its window. A loss-based sender then averages roughly (1,460 bytes / 80 ms) x 1.22 / sqrt(0.02), about 1.3 Mbit/s, so the same poster takes about 3 seconds. That is the Mathis estimate for Reno-style senders; the exact figure varies by algorithm, the lesson doesn't.

What an ACK Proves

Go back to Maya's tap, on a connection that is already open and warm. Both packets of "buy 14C" reach the load balancer, it acknowledges every byte, and that ACK comes home to her phone.

So is the seat hers?

Not yet, and maybe not at all. That ACK came from the load balancer's kernel, and it proves exactly one thing: the bytes are sitting in a receive buffer on that machine. No app server has necessarily read them. Nobody has charged her $40. The database hasn't written 14C to her name, and order 8812 doesn't exist yet. Any of those steps can still fail, and the ACK would look the same. An ACK is a promise about bytes. It says nothing about what anyone did with them.

This idea has a famous paper behind it. Saltzer, Reed and Clark, "End-to-End Arguments in System Design" (1984):

The function in question can completely and correctly be implemented only with the knowledge and help of the application standing at the end points of the communication system.

For tixdrop, the function is "sell 14C to Maya exactly once", and only the application with its database knows whether that happened. TCP's reliability is still worth having, because it makes the common case fast and simple, but it can never be the whole answer.

Here is the case it can't catch. Maya's request arrives, tixdrop sells her the seat, and the connection dies before the response gets back. Her phone opens a new connection and sends "buy 14C" again. To TCP that is brand new data on a brand new connection, with fresh sequence numbers from a fresh handshake. Nothing on any ruler says it is a repeat. Duplicate detection in TCP only works inside one connection.

Only the application can spot that duplicate. API design gives it the tool to do so: an idempotency key.

Connections in Practice

Keep Them Warm

Almost everything expensive about TCP happens at the start: the handshake, then the ramp while the window grows. So the cheapest connection is the one already open. If Maya's phone reuses the connection it opened for the seat map, "buy 14C" gets its answer in one round trip instead of two.

Browsers do this with keep-alive (persistent connections, the default since HTTP/1.1). Servers and services do it with connection pools: a set of open connections handed out for a request and put back afterwards. Database drivers, HTTP clients and message queue clients all pool.

Idle Timeouts

Every stateful box on the path will eventually close a connection nobody is using. tixdrop's load balancer closes any connection idle for 60 seconds, a common default. If a client's pool holds connections longer than that, the next request on a dead one gets a reset instead of an answer. The fix is boring and important: make the client's idle timeout shorter than the load balancer's.

transport := &http.Transport{
	MaxIdleConnsPerHost: 100,
	IdleConnTimeout:     50 * time.Second, // under the load balancer's 60 s
}

The same rule applies one hop further in, in the other direction. The load balancer pools connections to the app servers, so each app server's keep-alive timeout must be longer than the load balancer's idle timeout, or the load balancer will send a request down a connection the app server just closed. Node.js's default server keep-alive of 5 seconds behind a 60-second load balancer is a classic source of intermittent 502s.

Warm Is Not Always Fast

On Linux, a connection that sits idle for longer than its retransmission timeout drops back to slow start by default, on the theory that the path may have changed. A pooled connection that waited a minute can ramp from 10 packets again. For long-lived connections between services in one data center, some teams turn this off:

# 1 = restart slow start after idle (the default)
sysctl net.ipv4.tcp_slow_start_after_idle
 
# which congestion control algorithm is in use
sysctl net.ipv4.tcp_congestion_control
 
# live per-connection state: cwnd, rtt, rto, retransmits
ss -ti state established

Resets and Silence

A connection can end two ways, and they fail very differently:

• A reset. The other end (or a box in between) sends RST: "I have no such connection." Your next read or write fails at once with "connection reset by peer". Annoying, but fast.
• Silence. The other machine lost power, or a firewall forgot the connection. No RST comes. If you are sending, TCP retransmits with backoff; with Linux defaults (tcp_retries2 = 15) it gives up after roughly 15 minutes. If the connection is idle, TCP notices nothing at all until keepalive probes start, which by default is after two hours.

Neither of those is a timeout you want a checkout to wait on. Your application needs its own deadlines; the end-to-end anatomy and timeouts and retries lessons build them.

Connection State

Each open connection costs both ends memory for its state and buffers, and a file descriptor. At the drop, 40,000 fans hold connections to tixdrop's load balancer at once. Idle, a socket costs the kernel a few kilobytes; busy, its buffers can grow to megabytes. Size file descriptor limits (ulimit -n) and memory for the peak, not the average. On the other side of the load balancer, connections to one app server share a limited range of source ports (about 28,000 with Linux defaults), which is one more reason to reuse connections rather than open one per request. The load balancing lesson goes deeper into how load balancers handle this.

Head-of-Line Blocking and QUIC

There is one more cost, and it comes straight from the thing TCP promises: everything in order.

Say tixdrop sends Maya the seat map, the poster, and the response to "buy 14C" over one connection. That is exactly how HTTP/2 works: many streams multiplexed over one TCP byte stream. One poster packet is lost, and the purchase response arrives right behind it. TCP won't hand anything past the gap to her app until the lost packet is resent, so her purchase waits at least another round trip for a picture she doesn't care about. That is head-of-line blocking. On lossy café wifi it can make HTTP/2 slower than HTTP/1.1, which spreads requests across several connections.

QUIC (RFC 9000), the transport under HTTP/3, fixes this by running over UDP and keeping order per stream. A lost poster packet stalls only the poster. It keeps everything this lesson built (sequence numbers, ACKs, windows, congestion control) and draws the ordering line around each stream instead of the whole connection. It also folds the TLS handshake into its own, and lets a connection survive Maya's phone switching from café wifi to cellular. Streams still share one congestion window, so a lost packet slows every stream's sending rate; it just no longer blocks their delivery.

What's Next

The one sentence to keep: TCP turns a wire that loses, reorders and duplicates packets into an ordered stream of bytes, paid for in round trips, and it promises nothing about what the other end did with them.

Maya's "buy 14C" arrived whole: every byte, in order, exactly once on that connection. But look at what those bytes say:

POST /orders HTTP/1.1
Host: tixdrop.example
Cookie: session=...
Content-Type: application/x-www-form-urlencoded
 
seat=14C&amount=40&card=...

Her session cookie, the seat, $40, and her card number, all in plain readable text. Anyone on the same café wifi can read every byte, and can change them on the way. TCP has no encryption and no way to know who is on the other end.

So how do Maya's phone and tixdrop agree on a secret when everyone in the café can listen? That is TLS, and it's next.

References

• Transmission Control Protocol (TCP), RFC 9293
• TCP Congestion Control, RFC 5681
• Computing TCP's Retransmission Timer, RFC 6298
• Increasing TCP's Initial Window, RFC 6928
• TCP Extensions for High Performance, RFC 7323
• QUIC: A UDP-Based Multiplexed and Secure Transport, RFC 9000
• Van Jacobson and Michael J. Karels, "Congestion Avoidance and Control" (SIGCOMM 1988)
• J. H. Saltzer, D. P. Reed and D. D. Clark, "End-to-End Arguments in System Design" (1984)