Stacktrace#02nginx/nginx

How nginx Tells Time Without Asking

64 rotating slots, a memory barrier, and an atomic pointer swap. Millions of readers, zero locks.

Jan 2, 20266 min read

written byKarn

Deep in nginx's source code, there's a trick that lets millions of requests read the current time without ever waiting for each other. It involves 64 pre-allocated slots, a memory barrier, and a bet that your thread won't sleep for a full minute.

Every Response Needs a Timestamp

Every HTTP response nginx sends includes a Date header:

HTTP/1.1 200 OK
Date: Tue, 09 Dec 2025 14:30:00 GMT
Content-Type: text/html

That timestamp gets written to logs too. And checked against caches. A single request might need the current time three or four times.

nginx handles millions of requests per second. That's millions of "what time is it?" questions. Every. Single. Second.

Why Asking the OS for Time is Expensive

The obvious approach: call gettimeofday() and ask the operating system.

But gettimeofday() is a syscall. And syscalls are expensive:

1. Your program stops running
2. The CPU switches from "user mode" to "kernel mode"
3. The OS reads the hardware clock
4. The OS does some math
5. The CPU switches back to "user mode"
6. Your program resumes

Each syscall costs somewhere between 100 and 1000 nanoseconds. That sounds tiny. But at 10 million requests per second, with 3-4 time checks per request? You're burning 3-4 seconds of CPU time every second just asking what time it is.

Okay, so cache it. Store the time in a variable, update it once per second. Problem solved, right?

Not quite. Because now you have a new problem.

The Concurrency Problem

Let's say you cache the time in a global variable. One thread updates it every second. Millions of readers access it constantly.

What happens when a reader is halfway through copying the time value, and the writer updates it?

Writer: time = "14:30:00"
                ↓ writer starts changing to "14:30:01"
Reader:         ↓ reader copies "14:" so far
Writer: time = "14:30:01"
Reader:         ↓ reader copies "30:01"
Result: Reader got "14:30:01" - actually fine this time

In practice, most code paths just read a single value like sec or grab a pre-formatted string - both of which are effectively atomic on modern hardware. But the possibility of tearing exists, and defensive code can't assume how readers will evolve.

The traditional fix: locks.

lock();
time_t now = cached_time;
unlock();

Now the reader waits for the writer to finish. And the writer waits for readers to finish. And readers wait for each other.

At millions of requests per second, that waiting adds up. You've just created a bottleneck.

nginx's Solution: 64 Rotating Slots

The insight that makes nginx's approach work: don't update the slot readers are using - write to a different slot, then point readers there.

From src/core/ngx_times.c:

#define NGX_TIME_SLOTS   64
 
static ngx_time_t  cached_time[NGX_TIME_SLOTS];
static u_char      cached_http_time[NGX_TIME_SLOTS]
                       [sizeof("Mon, 28 Sep 1970 06:00:00 GMT")];

nginx pre-allocates 64 copies of everything:

• 64 time structures
• 64 pre-formatted HTTP date strings
• 64 pre-formatted log date strings

There's a pointer that tells readers which slot is "current":

volatile ngx_time_t *ngx_cached_time;

How the Writer Updates

Slot:    [0] [1] [2] [3] ... [63]
          ^
          |
     ngx_cached_time points here
     (readers are using this)

1. Writer moves to the next slot (slot 0 → slot 1)
2. Writer fills in slot 1 with the new time data
3. Writer issues a memory barrier (more on this in a moment)
4. Writer swaps the pointer to slot 1

Slot:    [0] [1] [2] [3] ... [63]
              ^
              |
     ngx_cached_time now points here
     (new readers use this)

The key: readers never see a half-written slot. By the time the pointer changes, the new slot is completely filled in.

How Readers Read

time_t now = ngx_cached_time->sec;

That's it. No lock. No waiting. Just dereference the pointer and grab the value.

Why 64 Slots?

Here's the only way this can go wrong:

1. A reader starts copying from slot 0
2. The reader gets preempted (OS pauses it to run something else)
3. The writer cycles through ALL 64 slots and comes back to slot 0
4. The writer starts overwriting slot 0
5. The reader wakes up and finishes copying - but now the data is corrupted

The comment in the source code says:

"thread may get corrupted values only if it is preempted while copying and then it is not scheduled to run more than NGX_TIME_SLOTS seconds"

But here's the nuance: nginx doesn't always update once per second. Depending on configuration - sub-second timers, millisecond caches for logging - updates can happen more frequently. The real safety property is:

# slots ≥ update rate × worst-case thread stall time

64 is a practical heuristic that works for typical configurations. If your thread sleeps long enough to wrap around the entire slot ring, a corrupted timestamp is the least of your problems.

The Memory Barrier: The Crucial Detail

Look at the writer's last two operations:

cached_time[1].sec = 1733749800;   // write the time data
ngx_cached_time = &cached_time[1]; // swap the pointer

You'd assume these run top-to-bottom. But compilers are sneaky.

Modern compilers reorder instructions aggressively to optimize performance. The compiler looks at these two lines and thinks: "These write to different memory addresses. I can reorder them!" It might emit assembly that swaps the pointer before the time data is fully written.

The disaster scenario:

1. Compiler reorders the pointer swap before the data write
2. A reader sees the new pointer, starts reading from slot 1
3. But the time data hasn't been written to slot 1 yet
4. Reader gets garbage

The fix:

cached_time[1].sec = 1733749800;
ngx_memory_barrier();              // STOP. Finish everything above first.
ngx_cached_time = &cached_time[1];

The memory barrier tells the compiler: "Don't reorder instructions across this line."

On x86, this compiles to:

__asm__ volatile ("" ::: "memory")

This is a compiler barrier-it generates no actual CPU instructions. It's just a note to the compiler saying "flush any cached values in registers and don't reorder memory operations across this point."

That's enough for x86, where the CPU already guarantees store ordering. On ARM or other weakly-ordered architectures, nginx uses platform-specific macros that emit real hardware fence instructions (like dmb or dsb). The portable ngx_memory_barrier() macro expands to whatever's needed for the target architecture.

Why This Matters Beyond nginx

This pattern-write to a shadow copy, then atomically swap a pointer-shows up everywhere in high-performance systems:

• Database page buffers: Write to a new page, then swap pointers
• Game engine double-buffering: Render to a back buffer, then swap
• Lock-free queues: Producers and consumers operate on different ends

nginx's time cache is a textbook example of the pattern in production code that runs on roughly a third of all websites.

One More Thing

While poking around this file, I found another gem. nginx implements its own ngx_gmtime() function to convert timestamps to dates. Why? Because the standard library's localtime() function isn't safe to call from signal handlers.

The comment explains:

"localtime() and localtime_r() are not Async-Signal-Safe functions, therefore, they must not be called by a signal handler, so we use the cached GMT offset value. Fortunately the value is changed only two times a year."

That last line made me smile. "Fortunately the value is changed only two times a year"-referring to daylight saving time. They're caching the timezone offset and trusting that it won't matter if it's briefly wrong during DST transitions.

Practical engineering at its finest.

What's the most clever concurrency trick you've seen in a codebase? I'd love to hear about it.

Source: nginx source code - src/core/ngx_times.c