Concurrency: when two threads share data

In tier 1 a thread was a unit the scheduler could run, and a process was the box that owned the address space. That distinction now starts to hurt. Two threads inside one process share the same data: the same heap, the same globals, the same counter. Nothing in the hardware stops them from touching it at the same instant, and the moment they do, the arithmetic you wrote is no longer the arithmetic the machine performs.

Start with the smallest possible example. Two threads, each incrementing one shared counter a million times. Every programmer will tell you the answer is 2000000. Run it and read the number.

/* counter.c */
#include <pthread.h>
#include <stdio.h>

static long counter = 0;

static void *worker(void *arg) {
  (void)arg;
  for (long i = 0; i < 1000000; i++) {
    counter = counter + 1;
  }
  return NULL;
}

int main(void) {
  pthread_t a, b;
  pthread_create(&a, NULL, worker, NULL);
  pthread_create(&b, NULL, worker, NULL);
  pthread_join(a, NULL);
  pthread_join(b, NULL);
  printf("%ld\n", counter);
  return 0;
}

/* $ gcc -O2 -pthread counter.c -o counter && ./counter
 * 2000000   <- never guaranteed
 * 1431927   <- typical instead
 */Two threads, one shared counter, no lock. Compile with gcc -O2 -pthread. The printed value changes from run to run.

The loop body looks like one action but is three: read the counter into a register, add one, write it back. A tick of the scheduler can land between any two of them, and with two cores the two threads do not even take turns. Here is the smallest schedule that loses an increment.

counter = 0

T1  r1 <- counter     r1 = 3417
T2  r2 <- counter     r2 = 3417
T1  r1 <- r1 + 1      r1 = 3418
T2  r2 <- r2 + 1      r2 = 3418
T1  counter <- r1     counter = 3418
T2  counter <- r2     counter = 3418

T1 done.  T2 done.  counter = 3418

Both threads read 3417, both add one, both write 3418. Two increments happened and the counter moved by one. This is a race condition: the result depends on the order the scheduler happened to pick, not on the program. On a real run of the program above you get something around 1.4 million instead of 2 million, and a different number every time. The bug is not rare and it is not random noise. It is a deterministic consequence of an interleaving you did not control.

The same loss happens everywhere the read-modify-write pattern appears: a bank balance, a reference count, an item appended to a list, a connection counter. What matters is not the operator. What matters is whether the value is read and then written back as two separate steps that another thread can slip between.

For a single counter there is a lighter answer. Processors have atomic read-modify-write instructions (lock xadd on x86, ldrex and strex on ARM), and the compiler exposes them as atomic operations. No lock object, no waiting thread, no separate critical section.

atomic_long counter = 0;

void *worker(void *arg) {
  (void)arg;
  for (long i = 0; i < 1000000; i++) {
    atomic_fetch_add_explicit(&counter, 1, memory_order_relaxed);
  }
  return NULL;
}

/* lock xadd   -> uncontended, a few nanoseconds
 * contended   -> the cache line is ping-ponged between cores
 */

What a lock costs depends entirely on contention. Uncontended, an atomic add is a few nanoseconds and a mutex acquire and release is a few tens of nanoseconds: a single instruction plus two barriers. The trouble starts when a second thread wants the same lock. Then the OS has to park that thread, which means a trip into the kernel, a context switch, and later a wake-up. That is microseconds, not nanoseconds - two to three orders of magnitude more than the work being protected. If the critical section is ten instructions long and four threads fight over it, the lock spends most of its time moving threads between cores, and the program gets slower as you add cores. This is why a growing counter is the classic false optimization: the atomic version scales until the cache line starts bouncing between cores, and then it stops too.

Now the uncomfortable part: why did the counter program pass your test? A race needs a particular interleaving, and interleavings are not equally likely. On one idle core, both threads run to completion in long stretches and the bad window is hit maybe once in a million loop iterations. On a busy machine, or with a debugger attached, or in production under load, that window is hit all the time. So the test passes on a laptop and the incident happens on a server with 32 cores. Concurrency bugs are timing-dependent, which means the absence of a failure is not evidence of correctness. Worse, adding a print statement to investigate can change the timing enough to hide the bug entirely.

The honest summary is that humans are bad at this, and the reason is specific. Reasoning about a sequential program means following one path. Reasoning about two threads means reasoning about the set of all interleavings, and their number grows combinatorially with the length of the code. Nobody holds that set in their head. This is why the practical answer is not to be careful but to narrow the surface: keep shared mutable state as small as possible, prefer immutable data passed between threads, use a message queue instead of shared memory where you can, and take a lock only around the few instructions that truly need it. Where the invariants cannot be made obvious, they must be written down and tested with tools built for it - sanitizers, model checkers and stress tests that deliberately vary timing - rather than with ordinary unit tests.