Service discovery: finding payments mid-drop

Maya's $40 charge for seat 14C is sitting inside tixdrop's orders service, ready to go. The code that sends it says one word, payments, because that is the service it wants to reach. The network doesn't know any names. It needs an address and a port, like 10.0.2.11:8443.

Turning a service name into an address that is alive right now is service discovery. On a quiet Tuesday it is easy. This lesson is about Friday at 10:00, in the middle of the drop, when payments grows from three machines to six and one of them dies within two minutes, and about the seconds when the answer is wrong.

Everything here happens inside tixdrop's Virginia data center:

A List in Config

The simplest way to turn the name into addresses is to write the addresses down:

# orders/config.yaml
payments:
  - 10.0.2.11:8443
  - 10.0.2.12:8443
  - 10.0.2.13:8443

That works for a while. Orders reads three addresses and spreads the charges across them.

At 10:01 the load climbs and the autoscaler starts three more payments machines, .14 through .16. They boot healthy and ready to charge cards, but orders' list still has three lines, so the new machines sit idle while the old three melt.

At 10:02 payments-2, the one at .12, dies. Nobody has edited the list, so orders keeps sending it a third of the charges, and every one of them fails. Fixing either problem means changing the config and restarting orders in the middle of the drop.

A fixed list goes wrong in both directions: it misses the machines that just arrived, and it keeps calling the ones that are gone. It is still the right tool for things that almost never move, like an outside API that already sits behind its own stable hostname.

A Name That Resolves

The first fix is to give payments a name and let one place keep track of the addresses behind it. That is what DNS does for the whole internet, and tixdrop's data center runs its own resolver for its services. When the autoscaler starts a machine, its address is added under the name. When a machine is removed, its address comes off. Orders asks where payments is and gets back all six addresses, the new ones included, without a restart:

$ dig payments.tixdrop.internal

;; ANSWER SECTION:
payments.tixdrop.internal.  30  IN  A  10.0.2.11
payments.tixdrop.internal.  30  IN  A  10.0.2.12
payments.tixdrop.internal.  30  IN  A  10.0.2.13
payments.tixdrop.internal.  30  IN  A  10.0.2.14
payments.tixdrop.internal.  30  IN  A  10.0.2.15
payments.tixdrop.internal.  30  IN  A  10.0.2.16

The 30 on each line is the answer's time to live (TTL): orders may keep reusing this answer for 30 seconds before it has to ask again. The TTL is what stops every service from asking the resolver on every single call. It is also a promise to be a little out of date, because for those 30 seconds orders hears about no change at all.

A few details worth knowing:

• Where the answer comes from. Inside a data center, orders asks a local resolver that is itself authoritative for internal names (on Kubernetes, that is CoreDNS). A browser looking up a public name asks a recursive resolver, which walks the chain from a root server to the .com servers to the domain's authoritative servers, caching each step for its TTL. Same idea, more layers of cache.
• Caches you didn't configure. Your language runtime, a sidecar or the HTTP client library may cache answers on top of the TTL, and some ignore the TTL completely. In Java, for example, the networkaddress.cache.ttl security property decides it. When you estimate staleness, find every cache between your code and the resolver.
• Ports. An A record carries only an address, so the port still lives in config. An SRV record carries host, port, priority and weight (_https._tcp.payments...), but a client only uses it if it looks SRV records up explicitly, and an ordinary HTTP client doesn't.

Who Noticed?

When payments-2 dies at 10:02, who notices, and takes its address off the name?

A health checker does. Every 5 seconds it asks each payments machine "are you okay?", and three misses in a row gets a machine removed. If payments-2 dies just after a good check, the next three checks fail at 5, 10 and 15 seconds, so it can take up to 15 seconds before anyone knows.

And a missed check can't say why it was missed. A machine that is gone and a machine that is too busy to answer look exactly the same from outside. That is the silence from the first lesson again, and the system models lesson takes it apart properly.

From here on, tixdrop runs on Kubernetes, which splits the health question in two:

• A readiness check asks "should I send you traffic?" Failing it takes the instance off the list and nothing else.
• A liveness check asks "are you stuck?" Failing it gets the container restarted.

# payments pod spec (excerpt)
readinessProbe:
  httpGet: { path: /ready, port: 8443, scheme: HTTPS }
  periodSeconds: 5
  failureThreshold: 3 # 3 misses: up to 15 s to notice
  timeoutSeconds: 1
livenessProbe:
  httpGet: { path: /live, port: 8443, scheme: HTTPS }
  periodSeconds: 10
  failureThreshold: 6 # a minute of silence before a restart

Keep them separate, and keep liveness loose. A busy machine should stop getting new charges, not get killed halfway through the ones it already has. And a liveness check that calls the database will restart every payments instance at once the first time the database is slow.

The Stale Window

Now add it up for payments-2. Detection takes up to 15 seconds. If orders fetched its DNS answer just before the address came off, it keeps using that answer for 30 more. That is up to 45 seconds in which orders still believes a dead machine is there. That gap is the stale window:

stale window ≈ detection time + propagation time (TTL, watch delay, every client cache)

During the drop orders sends about 20 charges a second. Pretend it spreads them evenly across all six instances: about 3.3 a second land on payments-2. Over 45 seconds that is about 150 charges sent to an address where nobody is listening. If Maya taps "buy" inside that window, there's a one in six chance hers is one of them.

What happens to those 150 depends on how payments-2 died. If the machine is up but the process is gone, the connection is refused within a round trip, and orders can safely try another address because the charge never got in the door. If the machine itself is gone, the connection attempt gets no answer at all and orders waits for its connect timeout. Either way nothing was charged, and a retry to a different instance is safe; API design sorts out which failures have that property.

Notice which piece is bigger. The TTL is twice the detection time, so it is the first knob to turn:

Past a point, a lower TTL only buys more lookups: once propagation is small, detection is the whole window.

Open Connections Ignore DNS

Turning the TTL down helps less than you'd think, because of how orders actually sends charges. It doesn't open a new connection for every charge; the TCP and TLS handshakes from TCP and TLS are too slow to repeat 20 times a second. It keeps a connection pool, opened at 9:59, and reuses those connections charge after charge.

Orders only looks the name up when it opens a new connection. So at 10:01, DNS lists six addresses while the pool still points at the old three. DNS steers new connections, not the ones already open, and even a TTL of 0 wouldn't change that. HTTP/2 and gRPC make it sharper: they send many requests over one long-lived connection, so a caller can go minutes without opening a new one.

The usual fix is on the server side: cap how long a connection may live (gRPC servers have a max connection age setting, and most proxies have an equivalent), so every caller reconnects, and looks the name up again, every few minutes.

A Registry That Pushes

Instead of orders coming back to ask, the list can come to orders the moment it changes. That is a service registry:

1. Register. When a payments machine starts, it tells the registry "I'm payments, at 10.0.2.14, port 8443."
2. Heartbeat. While it is alive it sends a heartbeat every 5 seconds; three missed heartbeats take it off the list.
3. Watch. Orders doesn't poll. It holds open a watch, a standing request that says "tell me as soon as this list changes," and the registry pushes the new list in about a second.

The stale window for payments-2 shrinks from 45 seconds to about 16, and the charges lost to it fall from about 150 to about 50. But look at what is left: nearly all of those 16 seconds are detection, and nothing can make a crashed machine announce that it is dead.

The registries you will meet:

• Kubernetes itself: the API server is the registry. Controllers keep an EndpointSlice per Service listing the ready pods, stored in etcd, and everything that routes traffic watches it. Applications use the Kubernetes API or DNS, not etcd directly.
• Consul: registration, health checks, and both DNS and HTTP interfaces for lookups and watches.
• ZooKeeper: instances create ephemeral nodes that vanish when their session stops heartbeating. Older Kafka versions depended on it; Kafka 4.0 removed it in favor of KRaft.
• Eureka: Netflix's registry, built for client-side discovery.

Who Picks the Instance?

With a live list in hand, someone still has to pick the machine that gets Maya's charge. That choice can live in two places.

Client-side discovery. Orders holds the list itself and picks an instance for every charge. There is no extra box in the path, but every caller needs that logic in every language you use. gRPC clients work this way, with one catch worth knowing: gRPC's default policy, pick_first, sends everything to one address until it fails, so spreading load means turning on round_robin or another policy.

Server-side discovery. Payments gets one stable virtual address, and orders sends everything there without seeing the list. Something behind that address knows who is ready and forwards the traffic. A separate load balancer box costs one more hop (about 0.5 ms inside the data center) and one more machine to keep alive. On Kubernetes, the virtual address is a Service with a ClusterIP, and rules on every node forward to the ready pods, so there is no extra hop. Those rules pick per connection, not per request, so pooled connections still stick where they first landed.

A service mesh moves the client-side picking out of your code. A small proxy, a sidecar, runs next to orders, watches the registry, and picks per request, in any language. Put one beside every service and you have a mesh, at the cost of an extra process on every machine and a little latency on every call.

For most teams the default is a Kubernetes Service, plus a mesh when you need per-request balancing. Two Service variants matter for discovery: a headless Service has no ClusterIP, so its DNS name returns the pod addresses directly and pod churn shows up in DNS answers; CoreDNS serves those with a 5-second TTL by default. An ExternalName Service just returns a DNS alias to something outside the cluster. How the chosen instance gets picked (round robin, least requests, and so on) is the subject of the load balancing lesson.

Leaving on Purpose

Machines also leave on purpose, and that is where plenty of teams drop requests without noticing.

At 10:30 the seats are long gone, and the autoscaler removes the three extra payments machines, starting with payments-4. Two things happen to it: it is told to shut down, and its address comes off the list. You would hope for that order, list first, then shutdown. The Kubernetes documentation says otherwise:

At the same time as the kubelet is starting graceful shutdown of the Pod, the control plane evaluates whether to remove that shutting-down Pod from EndpointSlice objects, where those objects represent a Service with a configured selector.

So payments-4 starts closing its doors while orders, about a second behind on the news, keeps sending it charges. At 10:30 that is a handful. But every deploy stops machines the same way, and during a drop each second of lag sends about 3 charges to a machine that is already hanging up.

Draining

The fix is to put the two steps back in order yourself. That sequence is draining:

1. Stop advertising. Come off the list while you can still serve perfectly well. On Kubernetes, a terminating pod is marked not ready in its EndpointSlice automatically; elsewhere, deregister from the registry or start failing readiness.
2. Wait. About 5 seconds, long enough for every caller, even the slow ones, to stop sending anything new.
3. Finish. Complete the charges already in flight and close pooled connections cleanly, so callers reconnect elsewhere instead of timing out.
4. Exit.

On Kubernetes the wait is a short preStop hook, which runs before the container gets its stop signal, plus a grace period long enough for the hook and the slowest charge:

spec:
  terminationGracePeriodSeconds: 30 # the default; covers the hook and in-flight work
  containers:
    - name: payments
      lifecycle:
        preStop:
          sleep:
            seconds: 5 # keep serving while every caller hears the news

When the stop signal (SIGTERM) arrives after the hook, payments stops accepting new connections, finishes what it has, and exits. Do that, and you can take machines away in the middle of a drop without a single charge landing on a closed door.

When the Registry Is Down

So far we've trusted the registry completely. But the registry is a distributed system too, and it can fail.

At 10:03 it stops answering, and it stays down for two minutes. Orders still has the list it fetched a second ago: five addresses, all alive, with payments-2 already gone. Should orders keep charging with that list, or stop until the registry is back?

Keep the list. It is a second old, and over two minutes a machine might die or a new one might start, but nearly every address on it will still be right. Stopping means no charges for two minutes, and at 20 a second that is 2,400 charges that don't happen, in the busiest two minutes of the year. A slightly stale list is mostly right; an empty list is an outage.

Keeping your last good copy when the source of truth goes quiet is called failing static. In practice: the client keeps the last list in memory, ideally on disk too so a restart during the outage still works, and a failed lookup never replaces a good list with an empty one.

Roblox learned what happens when everything depends on the registry being healthy. In October 2021 the site was down for 73 hours, and the postmortem sums it up in one line:

In short, the system failed because Consul was a single point of failure, and Consul was not healthy.

Too Eager to Remove

The opposite mistake is a discovery system that removes too much.

Say you require each payments health check to answer within 1 second, to be safe (1 second is also Kubernetes' default timeoutSeconds). At the peak of the drop every machine is so busy that its checks come back late. All six miss three in a row, and all six are removed. They were slow, not dead, but orders now has an empty list and nothing gets charged at all.

Meta hit exactly this in October 2021, at the scale of the internet. A backbone change cut off its data centers, and its DNS servers, which announce their routes using BGP, did what they were built to do:

To ensure reliable operation, our DNS servers disable those BGP advertisements if they themselves can not speak to our data centers, since this is an indication of an unhealthy network connection.

Every one of them withdrew at once, and for hours the rest of the internet couldn't find Facebook's address.

The defense is to distrust health checks when too many fail together. Envoy, a widely used proxy, has a panic threshold: if fewer than 50% of hosts look healthy (the default), it spreads traffic across all of them anyway, on the theory that a mass failure is more likely a broken check than a broken fleet. Two habits help too: give checks a timeout that fits a busy machine, and keep readiness about the instance itself, because a readiness check that fails when a shared dependency is slow removes every instance at the same moment.

The Takeaway

Discovery gives orders a recent guess about where payments is, not the truth. The config list, the DNS answer, the registry's watch and the pooled connection are all a little behind, so every call still needs a timeout and a plan for "that address was wrong."

Now Maya's $40 charge reaches a payments machine that is alive and well, and payments charges her card. It sends back "done," and that reply never arrives. Orders is left holding a timeout with no idea whether Maya paid. Can it just send the charge again? That is API design, and it's next.

References

• Service | Kubernetes
• Pod Lifecycle | Kubernetes
• Configure Liveness, Readiness and Startup Probes | Kubernetes
• Container Lifecycle Hooks | Kubernetes
• CoreDNS kubernetes plugin
• Envoy: Panic threshold
• Roblox Return to Service 10/28-10/31 2021
• More details about the October 4 outage (Engineering at Meta)
• Apache Kafka 4.0.0 Release Announcement