Secrets: keys, storage and rotation

The previous lesson ended on a password. Passwords are the secrets a human types; this lesson is about the secrets a program uses when nobody is watching: API keys, database passwords, signing keys, tokens for other services. They fail differently from human passwords. Nobody reuses a database password out of laziness, nobody forgets it, and nobody picks a weak one because it is easier to type. They fail because they are copied into places that were never meant to hold them, and because they live forever.

A secret committed to a repository is not a file you can delete. Git stores snapshots, and every snapshot that ever contained the key still contains it. Removing the line in a new commit removes it from the current state and changes nothing about the history. Even a force-push that rewrites the branch leaves the old objects reachable through the reflog, through any fork, through any clone that was already made, and through the storage of the provider for a while. This is not a Git flaw; it is what a content-addressed history is for. The practical conclusion is the one that costs teams the most time to accept: by the time you notice, the credential must be treated as public. Finding a leaked key in history and confirming it is still reachable. The search is the easy part.

$ git log --all -S 'AKIA' --oneline
9f2c1ab  remove keys from config
4d7e0f3  add deployment config

$ git show 4d7e0f3:config/prod.env | grep -c AKIA
1

$ git rev-list --all --count
8421

# the commit is still reachable: history is append-onlyFinding a leaked key in history and confirming it is still reachable. The search is the easy part.

Environment variables are the standard advice, and they are an improvement, but they are often oversold. What they genuinely buy: the secret is not in the repository, it is not in the container image, and it differs between staging and production without a code change. What they do not buy: any isolation at all. On Linux the environment of a running process is readable from /proc, so every process running as the same user can read the key. It ends up in crash dumps and in child processes you spawn, which is how a key from the payment service turns up in a log from an unrelated job. And on many platforms it is visible in the configuration screen of the dashboard, which is to say it is visible to everyone with deploy rights. Environment variables move a secret from a bad place to a better place. They are not a vault. An environment variable is readable by anything that can already run as that user. That is not nothing, but it is not secrecy. Key derivation is the bridge from a human secret to a cryptographic key. You do not use a password as an AES key directly, because passwords are short and low-entropy and keys must be uniformly random. A derivation function takes the password, a salt and a deliberately expensive cost parameter, and produces a key of the right length. The same idea appears in three places at once, and it is worth seeing that they are the same idea: deriving a key from a passphrase, deriving a password hash that resists guessing, and deriving a per-record key from a master key so that one leaked record key does not expose the rest. The cost parameter is what makes each of them resist an attacker who has the stored material. Disk encryption has a specific failure mode worth naming. If the key that unlocks the disk is stored on the same machine, or is fetched automatically at boot by a configuration file, then anyone who can boot that machine or read that file has the data. This is why full-disk encryption protects a stolen laptop and does almost nothing for a running server that an attacker has already reached.

in transit     TLS between client and service      a key you never see
at rest        disk encryption, KMS envelope      server holds the key
tokenized      provider keeps the card number    your database holds a token

PCI DSS 4.0 requires cardholder data encrypted in transit and at rest

None of the three protects against a valid account using the API.Three layers people mean when they say the data is encrypted. They protect against different things and none of them protects against a valid request.