Dapper, a Large-Scale Distributed Systems Tracing Infrastructure

Open almost any tracing tool and you will see one request drawn as a stack of bars against a clock. In our example shop, one checkout takes 1.2 seconds from the click to the answer. Under the top bar sit the calls it made: the cart, the payments service, a fraud check and a call to the bank. One bar is far longer than the rest: the bank call alone took a full second. OpenTelemetry's docs call this picture a waterfall diagram.

Five programs on five machines did that work, and none of them could see the others. So how does one picture of the whole request come together? And how do you keep it running in every service, all the time, without slowing anything down? Google answered both in a 14 page technical report published in April 2010, describing Dapper, the tracing system it had been running in production for over two years.

One Search, Thousands of Machines

The paper's example is web search. A front end sends your query to many hundreds of servers, each searching its own piece of the index. The same query also goes to systems for ads, spelling, images, videos and news, and the answers are combined into one page. Google called this universal search. Thousands of machines and many different services might be needed to answer one query.

This is the shape Dean and Barroso's The Tail at Scale describes, and one of its two authors, Luiz André Barroso, is on this one too. Now say searches get slow. An engineer looking only at the total latency may know there is a problem, but not which service is at fault, or why. The paper gives three reasons:

1. The engineer may not know which services are in use, because pieces are added and changed from week to week.
2. Nobody is an expert on the inside of every service, because each one is built and run by a different team.
3. Services and machines are shared by many clients, so the slowness might come from another application entirely.

So the job is to record all the work done in the system on behalf of one request, and tie it back to that request.