Talking to slow devices: blocking, interrupts and queues

A processor executes an instruction in a fraction of a nanosecond. A disk answers in a tenth of a millisecond. That is a gap of six orders of magnitude between the device the operating system manages and the thing doing the managing, and essentially every design decision in input and output exists to hide it.

CPU register: ~0.3 ns
L1 cache:     ~1 ns
RAM:          ~30 ns
SSD:          ~100 000 ns
HDD:          ~8 000 000 ns

SSD / RAM = 3000xThe gap the input and output subsystem exists to hide.

The simplest model is blocking input and output. The process calls read(), the kernel submits the request to the device, and the calling thread is taken off the run queue and put to sleep. It is not spinning and it is not scheduled: it costs nothing until the device finishes. Then the device raises an interrupt, the kernel wakes the thread, and read() returns with the data. From the point of view of the program a blocking read is one line of code. From the point of view of the system it is a context switch out and another one back, plus however long the device took.

The alternative to an interrupt is polling: the processor repeatedly asks the device whether it is done. That sounds obviously worse and it usually is, but not always. An interrupt is not free - the processor must stop what it is doing, save state, run the handler, and the handler destroys the usefulness of the instruction cache and the branch predictor. If a device completes a request every few microseconds, the interrupt rate becomes the bottleneck and the machine spends its time entering and leaving handlers. So the two are chosen by the timescale: interrupts for slow, unpredictable devices such as disks and keyboards, polling for fast ones where the wait is short and known. Network cards under heavy load are the classic case, which is why they switch from interrupts to polling automatically when the packet rate climbs.

polling:    10 000 000 spins/s, 1 core busy
interrupt:  no cost while the device is idle

slow device, 0.1 ms  -> interrupts win
fast device, 40 000 iops -> interrupts cost 4 ms/sThe tradeoff, in numbers. Neither answer is right in general; the timescale decides.

Now the cost that surprises people. A call to read() is not a function call. It is a controlled transition into the kernel: the processor switches to kernel mode, the registers are saved, arguments are validated, permissions are checked, and on return there is a check for pending signals and a possible reschedule. All of that happens before the kernel has read a single byte from anywhere. Measured on a modern processor, an empty system call costs between 100 and 300 nanoseconds, against roughly 2 nanoseconds for a call inside the same process. That is the number that explains a large fraction of real world performance problems.

size_t n = read(fd, buf, 4096);

call: ~2 ns
read(4096): ~200 ns
read(1): ~1000 ns

4096 calls instead of 1 -> 4000xThe same bytes delivered in one call and in four thousand calls.

Put those two numbers together and a familiar piece of advice stops being a style preference. A program that reads a 1 MiB file one byte at a time makes 1048576 system calls. At 200 nanoseconds each that is about 0.21 seconds of pure call overhead, before any data has been touched, and that is the optimistic case where every byte is already in the page cache. The same file read in 4 KiB blocks takes 256 calls and about 51 microseconds of overhead. Reading in 64 KiB blocks takes 16 calls. The ratio between the byte at a time version and the block version is measured in thousands, and none of it has anything to do with how fast the disk is.

When one thread must wait for many devices at once, blocking reads do not work: the first call blocks and the other nine sockets never get looked at. The old answer was select() and poll(), which hand the kernel a list of descriptors and ask which ones have data. Both are linear in the number of descriptors, so with ten thousand connections every call walks the whole list. epoll was the fix: register the descriptors once, and the kernel keeps the interesting ones on a ready list. The program then blocks once in epoll_wait and gets back only the sockets that actually have something.

io_uring was built to remove exactly that step. Instead of registering interest and then issuing a system call per operation, the program writes a request into a submission queue living in shared memory, and the kernel writes the result into a completion queue. One system call can submit a batch of dozens of reads, or, with the polling mode enabled, no system call at all. This is a completion model: a result appears in the completion queue when the read has already happened. The difference matters most when there are many small operations, because it turns N system calls into roughly one.

/* readiness model */
for (;;) {
  n = epoll_wait(ep, evs, 128, -1);
  for (i = 0; i < n; i++) read(evs[i].data.fd, buf, 4096);
}

/* completion model */
for (;;) {
  sqe = io_uring_get_sqe(&ring);
  io_uring_prep_read(sqe, fd, buf, 4096, 0);
  io_uring_submit(&ring);
}Two models side by side. The first asks whether it can read; the second asks for the read and collects the answer.

The practical difference between them is not philosophical. In the readiness model each event costs a system call for the wait plus a system call for the read. In the completion model the read is queued in memory and costs no transition at all. For a server doing tens of thousands of small operations per second that is the difference between spending most of its time in the kernel and spending most of it on actual work. It also explains why the numbers move so much with queue depth: one operation in flight can never fill a fast device.

epoll: 1 submit per batch
io_uring: 1 submit for N reads

queue depth 1: 1 000 000 iops
queue depth 32: 3 000 000 iopsThe same interface at different queue depths. Deep queues are what make fast devices useful.

Buffering is the general name for all the places data is copied. The kernel keeps a page cache, so a second read of the same file never reaches the disk. The C library keeps a user space buffer for stdio, so fgetc() usually does not call the kernel at all. Both of these are the same idea: batch the slow operation and amortize it. Both also have the same cost, which is that what you wrote is not necessarily what is on the device yet.

The practical rules that follow. Never issue a small operation when one big one would do: block size is the single biggest lever, and larger blocks win until they stop fitting in cache. Do not call fsync on every write unless correctness genuinely requires it, and then batch. When a thread must wait for many devices, use readiness or completion notification rather than one thread per connection, because a thread costs a stack and a context switch. And when measuring, measure the system time separately from the user time: a program whose system time dwarfs its user time is talking to the kernel far too often.