Clocks Are Broken: Physical, Logical, and Vector

Here's a question that sounds easy: which event happened first?

On a single machine, it's trivial. Your operating system keeps one clock, events go into a log in order, and you're done. But the moment you have two machines - two databases, two services in different data centers - you've got a real problem. There is no single global clock. Each machine has its own, and they will disagree.

This lesson is about time in distributed systems. Why you can't trust physical clocks, what logical clocks give you instead, and how vector clocks solve the problem that Lamport timestamps can't. By the end, you'll understand the tradeoffs behind every database's approach to event ordering.

The Ordering Problem

Picture two servers. One in New York, one in London. A user updates their email address, and both servers process the change. Server A stamps the event at 10:00:00.000. Server B stamps its event at 10:00:00.003 - three milliseconds later.

So Server A's write happened first... right?

Not necessarily. What if Server B's clock is running 5ms fast? That "later" timestamp might represent an event that actually happened before Server A's write. And if you're using last-write-wins to resolve conflicts, the wrong value survives. Silently. No error, no warning - just the wrong data.

The core problem: there is no single source of truth for time in a distributed system. Each machine keeps its own clock, and they drift apart constantly.

Physical Clocks: Why They Can't Be Trusted

Quartz Oscillators and Drift

Every server has a quartz crystal oscillator that vibrates to keep time. They're reasonably accurate - but they drift. A typical drift rate is 10-20 parts per million, which works out to roughly one second per day. Across a cluster of servers running for weeks, you can accumulate hundreds of milliseconds of drift.

NTP: The Fix That Doesn't Quite Fix It

The Network Time Protocol (NTP) synchronizes your server's clock to a reference time server. But the correction itself travels over the network, introducing delay. Best case, you get accuracy within a few milliseconds. Worst case, significantly more.

And here's the scary part: NTP can jump your clock backwards. One moment it's 10:00:05, the next it's 10:00:03. Your timestamps just went back in time. This actually happens in production systems.

Wall Clocks vs. Monotonic Clocks

There are actually two types of physical clock available to your code:

• Wall clock - answers "what time is it?" Can jump forward or backward due to NTP corrections. Good for displaying time to users, terrible for ordering events.
• Monotonic clock - answers "how long since X?" Always moves forward. Great for measuring durations on a single machine, but the values are meaningless across machines.

Neither clock type solves the fundamental problem of ordering events across machines. We need a different approach entirely.

Clock Drift in Practice: Last-Write-Wins Gone Wrong

Here's a concrete scenario showing how clock drift causes data loss:

1. Server A (accurate clock) writes X=1 at time t=100
2. Server B (clock is 5ms fast) writes X=2 - its clock says t=205, but real time is t=200
3. Server A writes X=3 at time t=300

A last-write-wins resolver sorts by timestamp: 100, 205, 300. Looks correct - X=3 wins. But what if Server A had written X=3 at real time t=201? Server B's write at "205" actually happened at real time 200, which is earlier. But the timestamp says it's later. The wrong value survives.

This isn't theoretical. Teams running Cassandra have hit this exact scenario. Clock skew causes silent data loss with last-write-wins resolution. No errors in your logs, no alerts - your data is just wrong.

Logical Clocks: Forget Real Time

The Happened-Before Relation

In 1978, Leslie Lamport published one of the most influential papers in computer science. His key insight: you usually don't need to know when something happened. You need to know the order. Specifically - did event A cause event B? Or are they independent?

Lamport defined the happened-before relation (written as →) with three rules:

1. Same process: If A happens before B on the same node, then A → B.
2. Message passing: If A is the sending of a message and B is the receiving of that message, then A → B. The send always precedes the receive.
3. Transitivity: If A → B and B → C, then A → C. Causality chains together.

If you can't establish a happened-before relationship between two events - no chain of local events and messages connects them - they're concurrent. Neither caused the other. And that's actually useful information, as we'll see.

Lamport Timestamps

Lamport took the happened-before relation and turned it into a simple counter. Each node maintains a single integer - its Lamport timestamp. The algorithm has just two rules:

• Local event: increment your counter (counter++)
• Receive message: set your counter to max(yours, message's) + 1

That's it. When a node sends a message, it includes its current counter value. When a node receives a message, it fast-forwards its counter if the message carries a higher value.

How It Works: A Three-Node Example

Consider three nodes - A, B, and C:

1. Node A does a local event, bumps to t=1, and sends a message to Node B
2. Node B receives the message, computes max(0, 1) + 1 = 2, sets its counter to 2
3. Node B does another event, bumps to t=3, and sends a message back to Node A
4. Node A receives it, computes max(1, 3) + 1 = 4
5. Meanwhile, Node C independently does local events at t=1 and t=2

The guarantee: if A caused B (through any chain of events and messages), then L(A) < L(B). The timestamp of the cause is always smaller than the timestamp of the effect.

Lamport's Limitation

Here's the catch, and it's a big one. The guarantee only works in one direction.

If A → B, then L(A) < L(B). ✅ Guaranteed.

If L(A) < L(B), then A → B. ❌ NOT guaranteed.

If A's timestamp is smaller than B's, you can't conclude that A caused B. Maybe it did. Maybe they're completely independent events and A just happened to have a smaller counter.

Consider a practical example: Node A does a local event (Lamport timestamp 1). Node B, completely independently, does a local event (also Lamport timestamp 1). Same number. Are they concurrent? Almost certainly. But you can't be sure from the timestamps alone.

In practice, this means if two clients write to the same key and you get timestamps 3 and 5, you might think "5 happened after 3, so 5 wins." But they could be completely independent writes, and you just silently dropped one client's update.

We need something stronger.

Vector Clocks

Vector clocks solve Lamport's limitation with one change: instead of a single counter, each node maintains a counter for every node in the system.

The Data Structure

If you have three nodes (A, B, C), each node maintains a vector of three numbers:

• Node A's clock: [A:1, B:0, C:0] - "I've seen 1 of my own events, nothing from B or C"
• Node B's clock: [A:0, B:1, C:0] - "I've seen 1 of my own events, nothing from A or C"
• Node C's clock: [A:0, B:0, C:1] - "I've seen 1 of my own events, nothing from A or B"

The Rules

1. Local event: increment YOUR entry only. [A:1, B:0, C:0] → [A:2, B:0, C:0]
2. Send message: increment your entry, attach the full vector to the message
3. Receive message: merge by taking the element-wise max of both vectors, then increment your entry

Detecting Concurrency: The Comparison Rules

This is where vector clocks become powerful. To compare two vector timestamps V1 and V2:

V1 ≤ V2 (happened-before): Every entry in V1 is ≤ the corresponding entry in V2. Example: [1, 2, 0] ≤ [1, 3, 1]. V1 is a causal ancestor of V2.

V1 || V2 (concurrent): Neither vector dominates - some entries in V1 are greater, some in V2 are greater. Example: [2, 1, 0] || [1, 2, 0]. These events are concurrent. Neither could have caused the other.

A Practical Example: Concurrent Document Edits

Alice edits a document on Node A. Her node's clock becomes [A:2, B:1]. Meanwhile, Bob edits the same document on Node B. His clock becomes [A:1, B:2].

Compare the vectors: A's entry is bigger in Alice's clock (2 > 1), but B's entry is bigger in Bob's clock (2 > 1). Neither vector dominates. The system knows these are concurrent edits.

Instead of silently picking a winner and losing someone's work, the system can present both versions for merging. That's exactly what Riak does, and what Amazon's original Dynamo paper described first.

With Lamport timestamps, you'd just pick whichever had the higher counter and lose one person's edit. Vector clocks give you the information to make a smarter decision.

The Tradeoff

Vector clocks use more space - O(n) per event, where n is the number of nodes. For a system with thousands of nodes, this becomes significant. Some systems use dotted version vectors or interval tree clocks to reduce the overhead while preserving concurrency detection.

Where They're Used in the Real World

The Original Dynamo Paper and Riak

The original Dynamo paper (2007) used vector clocks to detect conflicting writes across replicas. When concurrent writes happen, the system stores all versions and returns them to the application for resolution. More work for the developer, but you never silently lose data. Riak is the production database that most directly inherits this design and still exposes vector clocks to clients today.

A note on naming: Amazon's DynamoDB service, launched in 2012, is not the same system as the Dynamo paper. It shares the name and some heritage but uses a different architecture (single-leader replication per partition with MVCC), and does not expose vector clocks. We'll come back to this in lesson 14.

Google Spanner - TrueTime

Google went the opposite direction entirely. Instead of giving up on physical clocks, they invested in making them trustworthy. Every data center gets GPS receivers and atomic clocks. Their TrueTime API returns a confidence interval - "the real time is somewhere between earliest and latest" - and Spanner waits out the uncertainty before committing. If the interval is 7ms, it waits 7ms. Elegant, but it requires hardware most companies don't have.

Cassandra - Last-Write-Wins

Cassandra uses wall clock timestamps with last-write-wins resolution. It's fast and simple, but as we've seen, clock skew can cause silent data loss. Many teams have learned this the hard way. If you use Cassandra, make sure your NTP configuration is tight.

Consensus Protocols

Lamport timestamps (or similar logical clocks) are embedded in virtually every consensus protocol. Raft, Paxos, and their variants use logical sequence numbers to order log entries. You'll see them in action in the next two lessons on leader election and replication.

Summary

Every database you use has made a choice about how it handles time. Some trust physical clocks (Cassandra). Some use logical clocks for ordering (Raft, Paxos). Some use vector clocks for conflict detection (Riak, the original Dynamo paper). And Google built custom hardware to make physical clocks trustworthy (Spanner).

Now you know what each choice means - and what can go wrong.

Next up: you can order events now. But how do distributed nodes agree on who's in charge? That's Leader Election and Raft.