When something has already gone wrong

Everything so far has been about preventing things. This lesson is about the other half, which is what separates a security hobby from a security job: what happens between the moment something goes wrong and the moment it is over. It is a process lesson more than a technical one, and the process is the part that fails most often under pressure, because the correct action is frequently the one that feels least decisive. A post-mortem is the part of the process that makes the next incident cheaper, and it only works if people tell the truth. Blame and truth are in direct competition: the first question of an investigation that is looking for a culprit is whether anyone will volunteer information afterwards. The substitute for blame is not vagueness, it is a different question. Instead of asking who ran the wrong command, ask what made the wrong command look right, and what would have made the wrong outcome impossible rather than merely discouraged. An action item that says be more careful is not an action item. An action item that says the deploy script now refuses to run against production without an explicit confirmation flag is one, because it does not depend on anyone remembering anything. The same incident written up twice. Only the second version produces action items that survive contact with a busy week. The honest summary is that prevention and response fail in different ways. Prevention fails because a control you believed in did not hold, and it usually fails quietly. Response fails because of the things humans do under pressure, and it fails loudly: rebooting the machine that held the evidence, wiping the host because the symptoms stopped, restoring a backup taken after the intrusion began, or writing a report that ensures nobody reports the next one. The uncomfortable conclusion is that the outcome of an incident is decided less by the sophistication of the attacker than by whether the organisation had written down, and practised, what it would do.

The first and hardest stage is noticing. An event is anything observable; an alert is an event someone decided was worth attention; an incident is an alert someone decided was real. Each of those conversions can fail. Logs that are never reviewed are decoration. Alerts that fire four thousand times a day train everyone to ignore them, which is worse than having no alert at all, because it also removes the assumption that somebody is watching. The practical lesson is that detection capacity is not the number of tools but the number of signals a human will actually look at. Everything downstream in this lesson is unavailable to an organisation that has not solved this stage, because you cannot contain what you have not noticed. Rough industry picture of how long breaches go unnoticed. The variation between organisations is far larger than the variation between years. An alert needs three things to be useful: a description of what was observed, enough context to judge it, and a defined action. An alert that says only that something anomalous happened is worse than silence, because it costs attention and returns nothing. The baseline is what makes context possible. You cannot say a login at 03:14 is unusual unless you know what a normal hour looks like for that account, which means the useful work starts long before the incident, when somebody decides what to record.

$ sudo dd if=/dev/sda1 of=/mnt/evidence/disk.img bs=4M conv=noerror,sync
$ sha256sum /mnt/evidence/disk.img | tee /mnt/evidence/disk.img.sha256

$ date -u +"%Y-%m-%dT%H:%M:%SZ" | tee -a /mnt/evidence/chain-of-custody.txt

# every later look works on a copy, never on the original
$ cp /mnt/evidence/disk.img /tmp/work.imgCapturing a disk image and starting a chain of custody. The hash is the point: it is what lets you say later that the copy was not altered.

An incident is an event that threatens or violates policy, confidentiality, integrity or availability. A breach is a specific kind of incident: one where data was actually accessed or taken by someone who should not have had it. The distinction is not pedantry, it changes what you do. A phishing campaign that nobody clicked is an incident with no breach, and the response is to block the sender and warn people. A misconfigured bucket that was public for a week is potentially a breach even though nothing technical broke, and the response involves finding out whether anyone read it. Collapsing the two makes organisations either overreact to noise or underreact to a real exposure.

Containment stops the damage. Eradication removes the cause. Restoration puts the service back. These are three different goals and they are frequently run as one, which is where the trouble starts. The instinct under pressure is to reboot, reinstall and restore, because that makes the symptoms go away. A rebuild that starts before you understand how the attacker got in produces a machine that is clean and a mystery that is permanent, and if the entry path was not closed, the next build is compromised in the same way. The order matters, and so does the willingness to keep a compromised machine isolated and running for a while rather than wiping it immediately. The three stages and the action each one is allowed to take. Notice that preservation happens inside containment, before eradication wipes the machine. Restoring from backup is an excellent answer to a hardware failure and a dangerous answer to an intrusion. The question that decides it is not how good the backup is, but when the attacker arrived relative to when the backup was taken. If the backup was made after the intrusion began, whatever persistence mechanism was installed may still be in it, and you will restore the compromise along with the data. This is the reason that containment comes before restoration: you need enough understanding of the timeline to pick a restore point that predates the first sign of entry, and you need to have closed the entry path before you bring anything back online. Evidence has a lifetime, and it runs out in the opposite order from what most people expect. Anything in memory, in a running process, in a network connection or in a temporary file is gone the moment the machine loses power or reboots. Anything on disk survives a reboot but not a reinstall. Anything in a central log store survives both, if it was already shipped there, which is why centralised logging is a forensic control and not just an operational convenience. The first actions of a response are therefore almost mechanical: note the current time, capture what is volatile before it disappears, and only then start deciding what to do about the machine. The rule that follows from this is short and is the one people break most often in the first five minutes: do not reboot. A reboot clears memory, drops network connections, stops running processes and overwrites part of the state you needed, and the attacker may have left a startup entry that the reboot triggers. If the machine must be isolated, unplug the network rather than the power. This is unintuitive precisely because turning it off and on again is the correct answer to almost every other problem a computer has.