fsync and the loss window

Lesson 1 ended with a rule: when the broker answers OK to a send, the message must survive a restart. Lesson 2 built the file that keeps it. The broker appends each message to the end of a segment file, as a record that starts with its length and carries a checksum, a short number computed from the record's bytes. After the append, it answers OK.

Now run a sale through that broker. Orders arrive at 40 a second.

The file ends with the last record that was written before 14:00:00.2. Order 4127 is not in it. Nothing after it is in it either:

OKs lost = send rate x time since the last copy to the drive
         = 40/s x 20 s
         = 800 messages

The broker answered OK to all 800. It had appended every one of them to its file. They are gone.

Now the second half of the puzzle. Two weeks earlier, during another sale, a bug crashed the broker process. The machine and its operating system kept running. The broker restarted 2 seconds later and lost nothing at all. It was the same broker, the same code and the same write calls. One failure lost 800 promises. The other lost none.

By the end of this lesson you will know why. You will be able to state, as a number, how many OKs any broker design can lose, and to close that gap without making every send slow. You will also see how the broker finds the clean end of its file when the power comes back.

Where a Write Puts Your Bytes

A program cannot write to a drive directly. It asks the kernel, the core of the operating system, which controls the hardware. On Linux and macOS the request is a system call named write. Lesson 2 named the place the bytes go first: the page cache, memory inside the kernel that holds file data in fixed-size blocks called pages.

Here is what write does when the broker appends a record:

1. The kernel finds the page that holds the end of the file.
2. It copies the record's bytes from the broker's memory into that page.
3. It marks the page dirty: changed in memory, not yet on the drive.
4. It returns.

None of these steps touches the drive. That is why the call takes about 0.002 ms. The drive work happens later, on the kernel's own schedule. On Linux, kernel threads called flusher threads wake up every 5 seconds by default. Each time, they write out data that has been dirty for longer than 30 seconds. They also start earlier when too much memory is dirty.

longest normal wait on Linux defaults
  = time until the data counts as old + time until the next wake-up
  = 30 s + 5 s
  = about 35 s

There is one more layer below the kernel. Most drives have a write cache: memory inside the drive itself. A drive can report data as written while it still sits in that memory. Only later does it reach stable storage, the flash chips or magnetic platters that keep data without power.

So a record passes through four places on its way down:

The "usually" in the third row is there because some drives and disk controllers carry a battery or capacitor that keeps their cache alive long enough to save it. Many do not. A later section returns to this.

Two Failures, Two Results

Now replay both failures from the puzzle against that table.

The process crash. The bug killed the broker process. Its memory is gone. But the kernel kept running, and so did the page cache. All 800 records were in dirty pages, and the kernel wrote them to the drive later, as it always does. When the broker restarted and read its file, the kernel served those pages straight from memory. To the new broker process, the file was complete. Every record whose write call had returned was there.

The power loss. When the power failed, every kind of memory in the machine lost its contents at the same moment: the broker's, the kernel's, and the drive's cache. Only stable storage kept anything. The kernel had last copied this file to the drive at 14:00:00.2. Everything written after that existed only in memory, and it vanished. A kernel crash (a "kernel panic") has the same effect for this purpose: the machine stops, and the page cache goes with it.

So the word "written" has two meanings, and they protect against different failures:

• Written: the write call returned. The bytes are in the page cache. They survive the broker process dying.
• Durable: the bytes are in stable storage. They survive the whole machine losing power.

Between the two is a gap. A record that is written but not yet durable will be lost if the power fails now. The OKs the broker gave for records in that gap form the loss window: the promises that a power loss can break.

Look back at lesson 1's third break. An engineer redeployed the broker 80 seconds into a provider outage, and 3,200 waiting messages were lost. A redeploy stops the broker process. It does not stop the machine. So a broker that writes each message to its file before answering OK already keeps all 3,200. For the restart lesson 1 tested, "written" is enough. A power loss is a new and harder test, and it needs the second meaning.

fsync: Asking for the Second Meaning

The system call that turns "written" into "durable" is fsync. The broker passes it an open file. The Linux manual describes it this way: it "transfers ("flushes") all modified in-core data of" the file "to the disk device (or other permanent storage device)". "In-core" is an old word for memory.

In steps:

1. The kernel sends every dirty page of that file to the drive.
2. It asks the drive to move its write cache to stable storage.
3. It waits until the drive confirms.
4. It returns.

The call blocks the caller until step 3 is done. When it returns without an error, every byte written to that file before the call is durable. Bytes written while the call was running are not covered.

Three details matter for a broker.

A new file needs its directory flushed too. A file's name lives in its directory, which is a separate thing on disk. The Linux manual says: "Calling fsync() does not necessarily ensure that the entry in the directory containing the file has also reached disk." When the broker starts a new segment file, it must also call fsync on the directory. Otherwise a power loss can leave the data flushed but the file's name missing, and the broker would not find the segment.

fdatasync saves less than it seems for a log. A sister call, fdatasync, skips metadata that is not needed to read the data back, such as the time the file was last changed. But an append changes the file's size, and the manual lists the size as metadata that is needed. So for an append-only log, fdatasync still writes most of what fsync writes.

Order matters. If the broker keeps a small file that points into the log (for example "the log is valid up to byte 104,857"), it must make the log data durable before it writes the pointer. Otherwise a power loss can leave a pointer to bytes that never reached the drive.

When the Drive Says Yes Too Early

fsync is only as honest as the drive under it. Step 2 above asks the drive to empty its cache. Not every system asks, and not every drive obeys.

Apple's manual page for fsync on macOS says the call "will flush all data from the host to the drive", and then that "the drive itself may not physically write the data to the platters for quite some time and it may be written in an out-of-order sequence." For a real flush, macOS has a separate request, F_FULLFSYNC, which "asks the drive to flush all buffered data to permanent storage."

The PostgreSQL documentation warns about the drives themselves. It says "many solid-state drives (SSD) also have volatile write-back caches", that "many of these do not honor cache flush commands by default", and that "consumer-grade IDE and SATA drives are particularly likely to have write-back caches that will not survive a power failure." On Linux, fsync does ask the drive to flush its cache; the manual notes that older kernels and less common file systems did not know how.

So a broker can do everything in this lesson correctly and still lose data to a drive that reports a flush it did not do. The defences are outside the broker's code: use drives whose documentation says their cache survives power loss, and test the whole machine by cutting its power while it runs. The measurements in the next section show the size of this gap on one laptop.

When fsync Fails

fsync can return an error, for example when the drive is failing. The natural reaction is to try again. On Linux, that is a mistake.

The PostgreSQL project found this in 2018. Its notes on the problem explain that, after a failed write-back, Linux may mark the failed pages as clean, so a later fsync can report success for data that never reached the drive. In their words, "you still only get the error once (so retrying fsync() is not OK)." PostgreSQL's fix was to stop the whole database with a fatal error when fsync fails, and to recover from its log on restart.

The same rule fits a broker. When fsync fails, the broker no longer knows which of its records are durable. It must not answer OK to any send waiting on that flush. The safe move is to stop, restart, and run the recovery described later in this lesson.

What fsync Costs

Measure the cost instead of guessing it. This recipe appends 1 KB records to a file and times each flush. It is Python, and runs as python3 flush.py full 10:

import os, sys, time, fcntl, statistics
 
flush = os.fsync if sys.argv[1] == "fsync" else (lambda fd: fcntl.fcntl(fd, fcntl.F_FULLFSYNC))
batch = int(sys.argv[2])               # records written before each flush
record = os.urandom(1024)              # one 1 KB message
fd = os.open("segment.test", os.O_WRONLY | os.O_CREAT | os.O_APPEND, 0o600)
times = []
for _ in range(300):
    start = time.perf_counter()
    for _ in range(batch):
        os.write(fd, record)
    flush(fd)
    times.append((time.perf_counter() - start) * 1000)
print(f"median {statistics.median(times):.2f} ms per flush of {batch} KB")
os.close(fd); os.remove("segment.test")

Line by line:

• flush = ... picks the flush to time: the plain fsync call, or the macOS F_FULLFSYNC request that also empties the drive's cache. On Linux, use fsync, which already asks the drive to flush.
• batch is how many records to write before each flush.
• record is 1,024 random bytes, the size of one message.
• os.open(..., O_APPEND, ...) opens the file so that every write goes to the end, the way a segment grows.
• The loop runs 300 times. Each time it writes batch records with os.write (the write system call), flushes once, and stores how long that took in milliseconds.
• The last lines print the median, the middle value of the 300 timings, and delete the test file.

Measured on 2026-10-14 on a MacBook Pro with an Apple M1 Max, its built-in 1 TB Apple SSD, the APFS file system, macOS 27 and Python 3.14. Each row is the median of 300 flushes, from two runs. The first row comes from a separate loop of 5,000 write calls with no flush. Lesson 2's longer run of 100,000 appends measured a median of about 0.004 ms; both say the same thing: a write costs a few microseconds.

Three things stand out.

First, a durable flush is about 2,000 times slower than a write:

4.4 ms / 0.002 ms = 2,200

Second, the plain fsync on macOS costs 0.07 ms, and the full flush costs 4.4 ms. The difference is the drive emptying its own cache. That is the "drive says yes too early" gap from the previous section, measured: a broker that used plain fsync on this laptop would look 60 times faster and protect far less.

Third, and most important: the flush costs about the same whether it carries 1 KB or 40 KB. Most of the 4.4 ms is a fixed price for asking the drive to make its cache durable. The data itself adds little until the batch reaches about 100 KB.

These numbers are from one laptop. A server drive can be much faster or much slower. Run the recipe on your own hardware before you trust any number in this section, including these.

One fsync per Send

The simplest durable broker calls fsync after every record, then answers OK. Price it with the measured 4.4 ms.

A broker writes one segment through one writer, in order, as lesson 2 built it. So its flushes happen one after another:

most flushes per second = 1,000 ms / 4.4 ms = about 227

So one fsync per send caps one log at about 227 sends a second, no matter how fast the CPU or the network is.

At the shop's own rates, that is enough:

at 40/s:  time spent flushing each second = 40 x 4.4 ms = 176 ms   (the drive is busy 18% of the time)
          time added to each send         = about 4.4 ms
          checkout                        = 300 ms + 4.4 ms = about 304 ms
 
at 5/s:   time spent flushing each second = 5 x 4.4 ms = 22 ms     (2%)

Now push the rate. At 1,000 sends a second, the broker would need 4,400 ms of flushing every second. That is impossible. Sends would queue behind each other, and the wait for each OK would grow without limit. From lesson 7 on, the shop's queue carries 50,000 messages a second. One fsync per send cannot get close.

The measurements already contain the way out. Flushing 40 records cost barely more than flushing one.

Group Commit: One Flush for Many Sends

The idea: do not flush once per message. Flush once per moment, and let every message that arrived in that moment share the flush.

Here is the writer of the segment, with the exact steps:

the writer (one thread owns the segment file):
  loop:
    batch = every send waiting right now (wait if there is none)
    for each send in batch: append its record with write   # page cache, about 0.002 ms each
    fsync the segment                                       # about 4.4 to 6 ms
    for each send in batch: answer OK

While the writer is waiting inside fsync, new sends keep arriving. They wait in line. When the flush returns, the writer answers the old batch and takes everything in the line as the new batch. This is called group commit, a name from databases, where many transactions share one flush of the database's log.

Notice what the loop does not have: no timer and no batch size setting. The batch size adjusts to the load by itself.

sends that arrive during one flush = send rate x flush time
 
at 40/s:      40/s     x 0.0044 s = 0.18  -> almost every batch holds 1 send
at 1,000/s:   1,000/s  x 0.005 s  = 5     -> 5 KB per flush
at 10,000/s:  10,000/s x 0.005 s  = 50    -> 50 KB per flush, measured at about 5.3 ms

At low load, a send finds the writer idle and is flushed alone, with no extra wait. At high load, batches grow, and the number of flushes per second stays near the drive's limit of about 200:

What Group Commit Costs

• Every OK waits for a flush. A send that finds the writer idle waits one flush. A send that arrives right after a flush started waits for that flush to end, then for its own:

  best case:  writer idle when the send arrives       = 1 flush   = about 4.4 ms
  worst case: arrives right after a flush has started = 2 flushes = about 10 ms

  For checkout that means about 304 to 310 ms instead of 300 ms. With the OK sent after write, the disk added about 0.002 ms to each send. Now it adds milliseconds.

• A producer that waits for each OK is slow. If one connection sends a message, waits for OK, and only then sends the next, it can never go faster than one message per flush:

  one connection, one message at a time = 1 / 4.4 ms to 1 / 10 ms = about 100 to 227 a second

  To go faster, a producer must keep several sends in progress at once, or put many messages in one send. Checkout is fine: every customer's checkout is a separate request, so each one waits once and they all overlap.

• The writer is harder to build. It must hold every answer until its flush returns. It must never answer a send whose flush failed. And it must keep the order of records equal to the order of their OKs.

• Bytes still cost. Group commit moves the limit from "flushes per second" to "bytes per second". At 1,000 KB per flush, the measured time doubled to about 10 to 11 ms. Lesson 7 does the math for 50,000 messages a second, and finds that the number of fsyncs, not their size, runs out first.

Some brokers batch on a timer instead. The RabbitMQ documentation says its message store "persists messages to disk in batches after an interval (a few hundred milliseconds)" so as to make fewer fsync calls, and it sends the publisher's confirmation (its OK) only after the message is persisted. That makes fewer flushes when traffic is light, and the price is an OK that can take hundreds of milliseconds. Group commit without a timer flushes as soon as the previous flush ends, so its OK takes one or two flush times.

Choosing the Moment of OK

There are now four ways a broker can decide when to answer OK. Each one is a different promise.

One assumption sits under the "a process crash loses nothing" column: the record was passed to write before the OK. A broker that answers OK while the message is still only in its own memory, as NSQ does by default for up to 10,000 messages (lesson 1), loses those on a process crash too.

Look at rows two and four together, because they teach the central point of this lesson. Imagine a broker that runs exactly the group commit loop above, with the same flushes at the same times, but answers each OK right after the write instead of after the flush. Its cost on the drive is identical. Its promise is not:

at 10,000/s, flush time about 5 ms
 
OK after the flush:  OKs a power loss can erase = 0
OK before the flush: OKs a power loss can erase = 10,000/s x 0.005 s = 50

The schedule of fsync calls decides the cost. The moment of the OK decides the promise. A broker that flushes often but answers before the flush still has a window. A broker that answers after the flush has none, however rarely it flushes.

The Loss Window as a Number

Now the window can be stated precisely. It is the largest number of OKs that one power loss can erase:

loss window = send rate x longest time from an OK to the fsync that covers it
              (or a message count, if the design flushes by count and that comes first)

Work it for each design at the shop's rates.

OK after write, the kernel flushes on its own (Linux defaults):

longest time from OK to durable = 30 s + 5 s = about 35 s
 
sale peak:   40/s x 35 s = 1,400 OKs
normal day:   5/s x 35 s =   175 OKs

The opening puzzle lost 800. That was 20 seconds of a window that can reach 35.

OK after write, the broker flushes every 2,500 operations or every 2 seconds. These are NSQ's defaults, and the NSQ code later in this lesson shows them. NSQ counts both writes and reads toward the 2,500.

at 40/s in and 40/s out:
  operations per second   = 40 + 40 = 80
  time to reach 2,500     = 2,500 / 80/s = about 31 s   -> the 2 s timer fires first
  loss window             = 40/s x 2 s = 80 OKs
 
at 5,000/s in and 5,000/s out:
  operations per second   = 10,000
  time to reach 2,500     = 2,500 / 10,000/s = 0.25 s   -> the count comes first
  loss window             = 5,000/s x 0.25 s = 1,250 OKs

At low rates the window is measured in time (2 seconds). At high rates it is measured in messages (about 1,250 sends, half of the 2,500 operations).

OK after the group fsync:

loss window = 0 OKs

The broker still loses records in a power loss: the ones in the batch whose flush was running. But none of them had an OK. Checkout was still waiting. Its send times out, and by the producer's duty from lesson 1, it sends again. The retry may create a duplicate, and the consumer's emailed_orders table from lesson 1 already handles duplicates. A lost record that was never promised is not a broken promise.

Who pays for a window? The customers whose OKs fall inside it. With 1,400 lost OKs, 1,400 people like Asha are told "your email is on its way" and never get it, and no part of the system will ever retry.

A window is not always a mistake. A broker can accept a large window on each machine if something else covers it, and the usual cover is a copy of the message on other machines. That is Kafka's choice, later in this lesson. The broker in this course uses group commit. At the shop's rates it costs about 4.4 ms per send, against 300 ms of checkout, and it makes the window on one machine zero.

Finding the End of the Log After a Crash

The power is back. The broker starts. Before it can accept a single send, it must answer one question: where does the valid data in the last segment end?

The file's size does not answer it. The kernel writes a file in pages, and the drive writes pages in smaller pieces called sectors. The PostgreSQL documentation describes the result: sectors are "commonly 512 bytes", and a write "could fail due to power loss at any time, meaning some of the 512-byte sectors were written while others were not." Apple's manual adds that a drive "may also re-order the data so that later writes may be present, while earlier writes are not."

So the tail of the last segment can look three ways:

1. Clean. The file ends exactly after a whole record.
2. Short. The last record's length field promises more bytes than the file holds.
3. Damaged. The last record has its full length, but one of its sectors never landed. Part of the record is zeros, or old bytes from whatever was on the drive before.

A record caught in the middle like case 2 or case 3 is called a torn record.

The Scan

The broker walks the last segment from its start, record by record, and stops at the first record it cannot trust:

pos = 0
loop:
  if fewer than 16 bytes remain after pos: stop          (16 = the header, lesson 2)
  read length (4 bytes, ignoring the top bit, which flags attributes)
    and the stored CRC-32C (4 bytes) at pos
  if length < 12, or above the largest allowed record: stop
  if fewer than 4 + length bytes remain after pos: stop
  if CRC-32C of the length - 4 bytes after the checksum != stored: stop
  add index entries if due; pos = pos + 4 + length
cut the file at pos
fsync the file

The length counts every byte after itself, so a record of length 49, like Asha's, takes 4 + 49 = 53 bytes, and a length below 12 cannot even hold the checksum and the id. The length checks catch case 2. Only the checksum catches case 3. Picture a record of about 1 KB where the second 512-byte sector never reached the drive. The length field is correct, and the file holds every byte the length promises. Without a checksum the broker would hand a worker a message whose second half is zeros. The checksum computed over those bytes does not match the one stored when the record was whole, so the scan stops at the start of that record.

The same walk rebuilds the offset index from lesson 2, so that index never needs to be trusted after a crash. It is rebuilt from the records. The time index is the one exception, because the records store no send time. Entries it lost are written again with the time of the restart, so those messages look a little younger and are kept a little longer, never dropped early.

Why Cut Instead of Skip

Why throw away everything after the first bad record, instead of skipping that record and continuing?

Because a bad record's length cannot be trusted, and the length is the only map to where the next record begins. After a bad record, any byte could be the start of a record or the middle of one. Guessing would risk reading the middle of a message as a new message.

And with group commit, cutting is safe. Follow the order of events:

• Records are appended in order, and every fsync covers every byte written before it.
• A record gets its OK only after an fsync that covers it returns.
• So if a record is torn, the fsync covering it never returned, and no record after it was covered either.

Therefore every record after the first torn one was never answered with OK. The cut can drop messages, but only messages whose producers are still waiting, or will retry.

Here is order 4127's segment after a power loss under group commit:

Order 4128 survives even though it never got an OK. Checkout will send it again, and the queue will hold two messages for order 4128. That is the at-least-once rule from lesson 1 once more: recovery may keep a message that never got an OK, but it must never drop one that did.

Under the "OK after write" designs, the same cut drops records that did get an OK. That is the loss window, made physical.

Scanning Only What Might Be Torn

A log can hold many gigabytes. Scanning all of it on every restart would be slow. It is also unnecessary.

When a segment fills up and the broker starts the next one, it flushes the full segment. From then on, that segment is known to be durable. So the broker keeps a recovery point: the position up to which everything is known to be flushed. After a crash, it scans only from the recovery point to the end. Recovery time grows with the data written since the last known flush, not with the size of the log. Kafka, shown below, writes its recovery point to a small file every 60 seconds by default.

Small files like that one need their own care. The safe pattern has three steps: write the new contents to a temporary file, fsync it, then rename it over the old file. On POSIX file systems, a rename replaces the old name in one step. After a crash, the broker finds either the complete old file or the complete new one, never half of each. NSQ's disk queue does exactly this, shown below.

What about messages that were in flight at the crash? Lesson 1 already answered that: the broker treats every message that was not deleted as ready, and at-least-once delivery absorbs the duplicates. Lesson 4 builds the in-flight table in detail.

What fsync Cannot Save

fsync makes an OK survive a power loss on the machine that gave it. That is all it does. It cannot help when:

• the drive fails. Durable data on a dead drive is gone.
• the machine is lost, for example after a hardware fault, or when the disk of a virtual machine is deleted with it.
• the drive does not keep its promise, as in the section on drives that say yes too early.

Every one of these needs the same answer: a copy of the message on another machine before the OK. That is lesson 8, Surviving a dead machine.

Some brokers make the opposite bet: they skip fsync on the write path and rely on those copies instead. Kafka's documentation says "durability in Kafka does not require syncing data to disk", because a failed machine can recover the data from the copies on other machines. It adds: "We generally feel that the guarantees provided by replication are stronger than sync to local disk." The bet holds as long as the machines holding the copies do not all lose power together. If they share one power feed and it fails, every copy was in a page cache, and every copy is gone. That is the price of the bet, and lesson 8 shows how brokers spread copies so that one failure cannot take them all.

How NSQ and Kafka Chose

Two real brokers sit at two different points on this map. NSQ flushes on a schedule and answers before the flush. Kafka does not flush on the write path at all, and leans on copies. Both scan their logs after a crash, in very different ways.

NSQ: OK First, fsync Every 2,500 Operations or 2 Seconds

NSQ is an open-source broker written in Go, under the MIT license. Its disk storage lives in a separate library, go-diskqueue, also MIT. The excerpts below are from go-diskqueue at commit ff2a02e.

Recall from lesson 1 that NSQ, by default, keeps up to 10,000 messages per topic and per channel in memory and only sends the overflow to disk. The code below runs for messages that go to disk, or for every message if you start nsqd with --mem-queue-size=0.

When a message goes to disk, NSQ calls Put:

func (d *diskQueue) Put(data []byte) error {
	d.RLock()
	defer d.RUnlock()
 
	if d.exitFlag == 1 {
		return errors.New("exiting")
	}
 
	d.writeChan <- data
	return <-d.writeResponseChan
}

diskqueue.go lines 165 to 175.

Line by line:

• d.RLock() and defer d.RUnlock() hold a shared lock for the whole call, so the queue cannot be closed halfway through it.
• if d.exitFlag == 1 refuses the write if the queue is shutting down.
• d.writeChan <- data hands the message bytes to the queue's single I/O loop. A Go channel is a pipe between two concurrent functions; this one carries writes.
• return <-d.writeResponseChan waits for the I/O loop's answer and returns it. When it returns no error, nsqd answers the producer OK.

Now the I/O loop. It is one goroutine (Go's lightweight thread), so all reads, writes and flushes of the queue happen one at a time, in one place:

	syncTicker := time.NewTicker(d.syncTimeout)
 
	for {
		// dont sync all the time :)
		if count == d.syncEvery {
			d.needSync = true
		}
 
		if d.needSync {
			err = d.sync()
			if err != nil {
				d.logf(ERROR, "DISKQUEUE(%s) failed to sync - %s", d.name, err)
			}
			count = 0
		}
		// ...
		case dataWrite := <-d.writeChan:
			count++
			d.writeResponseChan <- d.writeOne(dataWrite)
		case <-syncTicker.C:
			if count == 0 {
				// avoid sync when there's no activity
				continue
			}
			d.needSync = true

diskqueue.go lines 663 to 716. The // ... replaces the code that reads the next message for consumers and the start of the select statement that waits for the next event.

Line by line:

• syncTicker := time.NewTicker(d.syncTimeout) creates a timer that fires every syncTimeout. nsqd sets it from --sync-timeout, default 2 seconds.
• for { is the loop the goroutine runs for its whole life. Each turn handles one event.
• if count == d.syncEvery checks how many operations happened since the last flush. nsqd sets syncEvery from --sync-every, default 2,500. When the count is reached, needSync is set.
• if d.needSync runs the flush: d.sync(). If it fails, the error is logged. Either way, count goes back to 0.
• case dataWrite := <-d.writeChan: runs when Put has sent a message.
• count++ counts it. In the cut part of the loop, each message handed to a consumer also does count++, so the 2,500 counts reads and writes together.
• d.writeResponseChan <- d.writeOne(dataWrite) appends the record with a write call, and sends the result straight back to Put.
• case <-syncTicker.C: runs every 2 seconds. If nothing happened (count == 0), it skips the flush. Otherwise it sets needSync, so the next turn of the loop flushes.

The line that decides the promise is d.writeResponseChan <- d.writeOne(dataWrite). The answer goes back to Put, and from there to the producer, right after the write call. The flush comes later, at the next 2,500th operation or the next tick of the 2 second timer. NSQ is the second row of the table in "Choosing the Moment of OK", and its loss window is the one worked out earlier: 80 OKs at 40 a second, about 1,250 at 5,000 a second.

sync itself (lines 443 to 460) calls writeFile.Sync(), Go's fsync, on the data file first. Only then does it call persistMetaData (lines 513 to 538), which saves the read and write positions with the safe pattern: write a temporary file, fsync it, rename it over the old one. The data is made durable before the pointer to it. NSQ also flushes every time it starts a new segment file (lines 386 to 390).

Two details connect back to earlier sections.

What happens when sync fails. If writeFile.Sync() returns an error, sync closes the data file, sets it to nil and returns the error. The loop logs it and carries on, and needSync is still set. On the next turn, sync runs again, finds no open data file to flush, and writes only the metadata. That is the retry the section on fsync errors warned about: after a failed flush, a later success does not prove the earlier data reached the drive.

NSQ's records have no checksum. NSQ's record is a 4-byte length followed by the message. On restart, NSQ trusts the saved write position. If the data file is longer than that position, the tail was written after the last flush, so NSQ starts writing a new file and lets the reader "salvage what it can" from the old one (lines 486 to 507). When the reader meets an impossible length, it renames the whole file with a .bad suffix and skips the rest of it (lines 326 to 332 and 610 to 646). That catches the "short" and "impossible length" tails. It cannot catch a "damaged" record whose length is intact: the disk queue has nothing to check the body against.

Kafka: No fsync on the Write Path, a CRC Scan on Recovery

Apache Kafka is an open-source log-based broker written in Java and Scala, under the Apache License 2.0. The excerpts below are from commit 03e6606.

Kafka's flush settings and their defaults:

    public static final String LOG_FLUSH_INTERVAL_MESSAGES_CONFIG = ServerTopicConfigSynonyms.serverSynonym(TopicConfig.FLUSH_MESSAGES_INTERVAL_CONFIG);
    public static final long LOG_FLUSH_INTERVAL_MESSAGES_DEFAULT = Long.MAX_VALUE;
    public static final String LOG_FLUSH_INTERVAL_MESSAGES_DOC = "The number of messages accumulated on a log partition before messages are flushed to disk.";
    // ...
    public static final String LOG_FLUSH_SCHEDULER_INTERVAL_MS_CONFIG = LOG_PREFIX + "flush.scheduler.interval.ms";
    public static final long LOG_FLUSH_SCHEDULER_INTERVAL_MS_DEFAULT = Long.MAX_VALUE;
    public static final String LOG_FLUSH_SCHEDULER_INTERVAL_MS_DOC = "The frequency in ms that the log flusher checks whether any log needs to be flushed to disk";
 
    public static final String LOG_FLUSH_INTERVAL_MS_CONFIG = ServerTopicConfigSynonyms.serverSynonym(TopicConfig.FLUSH_MS_CONFIG);
    public static final String LOG_FLUSH_INTERVAL_MS_DOC = "The maximum time in ms that a message in any topic is kept in memory before flushed to disk. If not set, the value in " + LOG_FLUSH_SCHEDULER_INTERVAL_MS_CONFIG + " is used";

ServerLogConfigs.java lines 100 to 113. The // ... replaces an unrelated setting for how long to wait before deleting a file.

Line by line:

• The first three lines define log.flush.interval.messages: flush a partition's log after this many messages. Its default is Long.MAX_VALUE, the largest number a Java long can hold.
• The next three define log.flush.scheduler.interval.ms: how often a background thread checks whether any log needs a flush. Its default is also Long.MAX_VALUE milliseconds.
• The last two define log.flush.interval.ms: the longest a message may stay in memory before a flush. The description says that when it is not set, the scheduler interval is used, so it inherits the same default.

How long is Long.MAX_VALUE messages at the shop's sale peak?

Long.MAX_VALUE                 = 9,223,372,036,854,775,807 messages
at 40/s                        = 9.22 x 10^18 / 40 = 2.3 x 10^17 seconds
in years (31,557,600 s a year) = about 7.3 billion years

In practice, both defaults mean "never". The append path checks the message setting in one place: after each append, if (localLog.unflushedMessages() >= config().flushInterval) flush(false); (UnifiedLog.java line 1300). With the default, that condition is never true.

Kafka does call fsync, only not before the OK. When a segment fills up and Kafka starts a new one, it schedules a background flush of everything before the new segment (UnifiedLog.java lines 2237 to 2241). That flush moves the recovery point forward, and Kafka saves the recovery point to a file every 60 seconds by default (ServerLogConfigs.java lines 115 to 117).

The documentation explains the choice: "We recommend using the default flush settings which disable application fsync entirely." Durability comes from copies on other machines, as the previous section quoted. The documentation also gives a cost reason: "fsync in most Linux filesystems blocks writes to the file", while the kernel's own background flushing locks much less. And it states what Kafka does after a crash: for any segment not known to be flushed, "Kafka will check the integrity of each message by checking its CRC" and rebuild the offset index. CRC is a widely used kind of checksum. Here is that scan:

    public int recover(ProducerStateManager producerStateManager, LeaderEpochFileCache leaderEpochCache) throws IOException {
        offsetIndex().reset();
        timeIndex().reset();
        txnIndex.reset();
        int validBytes = 0;
        int lastIndexEntry = 0;
        maxTimestampAndOffsetSoFar = TimestampOffset.UNKNOWN;
        try {
            for (RecordBatch batch : log.batches()) {
                batch.ensureValid();
                ensureOffsetInRange(batch.lastOffset());
                // ...
                // Build offset index
                if (validBytes - lastIndexEntry > indexIntervalBytes) {
                    offsetIndex().append(batch.lastOffset(), validBytes);
                    timeIndex().maybeAppend(maxTimestampSoFar(), shallowOffsetOfMaxTimestampSoFar());
                    lastIndexEntry = validBytes;
                }
                validBytes += batch.sizeInBytes();
                // ...
            }
        } catch (CorruptRecordException | InvalidRecordException e) {
            LOGGER.warn("Found invalid messages in log segment {} at byte offset {}.", log.file().getAbsolutePath(),
                validBytes, e);
        }
        int truncated = log.sizeInBytes() - validBytes;
        // ...
        log.truncateTo(validBytes);
        offsetIndex().trimToValidSize();

LogSegment.java lines 482 to 523. The cuts remove the tracking of the largest timestamp, the tracking of leader epochs and producer state (used by lesson 8's replication), and a debug log line.

Line by line:

• offsetIndex().reset() and the two lines after it throw away the segment's indexes. They are rebuilt below from the records themselves. The offset index's own source comment says "No attempt is made to checksum the contents of this file, in the event of a crash it is rebuilt."
• int validBytes = 0 is the number of bytes known to be good. It is the pos of the scan pseudocode.
• for (RecordBatch batch : log.batches()) walks the segment. Kafka stores records in batches, each with its own header and checksum. If the last batch is shorter than its length field says, the walk ends quietly; if the length is impossibly small, it throws (FileLogInputStream.java lines 64 to 94). These are the scan's length checks.
• batch.ensureValid() recomputes the batch's CRC-32C and compares it with the stored value. On a mismatch it throws CorruptRecordException (DefaultRecordBatch.java lines 150 to 159). This is the checksum check, and the only one that catches a "damaged" batch.
• ensureOffsetInRange(batch.lastOffset()) checks that the batch's offset fits the segment's index, which stores offsets relative to the segment's first offset.
• The if block adds an offset index entry each time more than indexIntervalBytes bytes (default 4,096) have passed since the last one. This is lesson 2's sparse index, rebuilt during recovery.
• validBytes += batch.sizeInBytes() moves the good mark forward. It runs only after the batch passed its checks.
• catch (CorruptRecordException | InvalidRecordException e) ends the walk at the first bad batch and logs where it was found.
• int truncated = log.sizeInBytes() - validBytes is how many bytes after the last good batch will be dropped.
• log.truncateTo(validBytes) cuts the file there.
• offsetIndex().trimToValidSize() shrinks the rebuilt index to the entries it actually holds.

Which segments does Kafka scan? Only the ones that might be torn. After an unclean shutdown, it recovers only segments at or after the recovery point. If one of them had to be cut, it deletes every later segment, because nothing after a torn batch can be trusted (LogLoader.java lines 456 to 491). That is the "cut instead of skip" rule, applied across files.

Side by Side

NATS JetStream is another open-source broker, written in Go. Two of its cells come from its file store: the hash check that cuts the file at the first bad record (filestore.go lines 1782 to 1803), and the relaxed "sync always" setting when a stream keeps copies on several servers (lines 686 to 701). Kafka and JetStream make the same bet: once there are copies, each machine's own fsync matters less.

The Takeaway

1. "Written" has two meanings. After write returns, the bytes are in the kernel's page cache and survive the broker process crashing. After fsync returns, they are on stable storage and survive the machine losing power.
2. The loss window is the number of OKs one power loss can erase: send rate times the longest time from an OK to the fsync that covers it. With the kernel flushing on its own, that is up to about 35 s on Linux defaults, 1,400 OKs at 40 orders a second. With NSQ's defaults, it is 2 s or 2,500 operations. With an OK after the flush, it is 0.
3. A durable flush is expensive (4.4 ms measured on one laptop, about 2,000 times a write) but nearly the same price for 1 KB or 40 KB. One fsync per send caps a log at about 227 sends a second. Group commit lets every send that arrives during a flush share the next one, so throughput grows with load while each OK waits one or two flush times.
4. The fsync schedule sets the cost. The moment of the OK sets the promise. A broker that flushes often but answers before the flush still has a window.
5. After a crash, the broker scans from its last known flush to the end of the log, checks each record's length and checksum, and cuts the file at the first record it cannot trust. With OK after the flush, every record past that point never got an OK, so the cut never breaks a promise.
6. fsync cannot survive a dead drive or a lost machine, and it cannot fix a drive that reports a flush it did not do. Only a copy on another machine can. Some brokers, like Kafka, skip fsync and rely on those copies instead.

The one sentence to keep: an OK is only as strong as the last thing that happened before it, so decide which thing that is, and count what can still be lost.

Our broker now keeps every OK through a power loss, at about 4.4 ms per send. The next problem is on the other side of the queue: a worker receives a message, and the broker must track it until the worker deletes it, gives up, or dies. That is Receiving without losing a message.

References

• fsync(2), Linux manual page (man7.org): what fsync flushes, the directory entry warning, fdatasync and the file size, older kernels that could not flush disk caches
• fsync(2), Apple developer manual page: fsync stops at the drive, the drive may reorder writes, F_FULLFSYNC
• Linux vm sysctl documentation (kernel.org): dirty_expire_centisecs and dirty_writeback_centisecs
• Linux mm/page-writeback.c at v6.12, lines 102 to 109: default write-back interval 5 * 100 and expiry 30 * 100 centiseconds
• PostgreSQL 18 documentation, Reliability: volatile drive write caches, SSDs that do not honour flush commands, 512-byte sectors and partial writes
• PostgreSQL wiki, Fsync Errors: why retrying a failed fsync is not safe on Linux, and PostgreSQL's change to stop on failure
• Apache Kafka 4.3 documentation, Hardware and OS: "Application vs. OS Flush Management", the recommendation to disable application fsync, durability through replication, CRC checks on recovery
• RabbitMQ documentation, Consumer Acknowledgements and Publisher Confirms: confirms after persisting to disk, batched fsync every few hundred milliseconds, quorum queue confirms
• NATS documentation, JetStream configuration: sync_interval default 2 minutes, always
• NSQ nsqd documentation: --sync-every (default 2500), --sync-timeout (default 2s), --mem-queue-size (default 10000)
• NSQ features and guarantees (NSQ documentation): "not durable (by default)"
• go-diskqueue repository, commit ff2a02e5299f130a68a7ffa6d200b9bb55671b73, MIT license (LICENSE)
  • diskqueue.go L165-L175: Put waits for the I/O loop's answer
  • diskqueue.go L663-L716: the I/O loop, syncEvery, the sync timer, and the answer sent before the flush
  • diskqueue.go L443-L460: sync, data first, then metadata
  • diskqueue.go L513-L538: persistMetaData, temporary file, fsync, rename
  • diskqueue.go L386-L390: a flush every time a new file starts
  • diskqueue.go L486-L507: recovery trusts the saved write position
  • diskqueue.go L326-L332 and L610-L646: an impossible length marks the file .bad
• NSQ repository, commit 85cf10c09c6c3c86160d6f0eb156f62d0efc1648, MIT license: nsqd/options.go L146-L149, defaults for MemQueueSize, MaxBytesPerFile, SyncEvery and SyncTimeout
• Apache Kafka repository, commit 03e66064307d921ae62a80332ae7bfca2d361c54, Apache License 2.0 (LICENSE)
  • ServerLogConfigs.java L100-L113: flush settings, both defaults Long.MAX_VALUE
  • ServerLogConfigs.java L115-L117: recovery point checkpoint every 60 s
  • UnifiedLog.java L1300: the only flush check on the append path
  • UnifiedLog.java L2237-L2241: background flush of the old segment when a new one starts
  • LogSegment.java L482-L523: recover, CRC check per batch and truncation
  • FileLogInputStream.java L64-L94: a short last batch ends the walk, an impossible size throws
  • DefaultRecordBatch.java L150-L159: ensureValid compares stored and computed CRC
  • LogLoader.java L456-L491: recover only segments after the recovery point; delete everything after a cut
• NATS server repository, commit 64dda80fdd87b41b2afcdc5faf5d846db6c189b0, Apache License 2.0
  • server/filestore.go L1782-L1803: per-record hash check and truncation on recovery
  • server/filestore.go L686-L701: "sync always" relaxed when a stream has copies
  • server/filestore.go L339: defaultSyncInterval = 2 * time.Minute
• Measurements: author's run on 2026-10-14, MacBook Pro (Apple M1 Max), built-in 1 TB Apple SSD (APPLE SSD AP1024R), APFS, macOS 27, Python 3.14, using the recipe in this lesson