Debugging instead of guessing

Troubleshooting a network is not about knowing more commands. It is about replacing guesses with a sequence of questions whose answers cannot be argued with. The method has three rules. Start with the check that is cheapest in time and rules out the most. Change exactly one thing at a time. And after every test, ask what the result proves and what it does not prove — a test that could have passed for two different reasons has told you nothing.

Cheapest first means: before you touch a configuration, ask whether the thing works at all from here, and whether it works from somewhere else. Comparing two vantage points costs one command and splits the problem space in half. Asking the user to try a different network, or running the same request from a different machine, is usually worth more than an hour of reading logs.

The ladder of questions, from the bottom: does the machine have an address at all, and a default route? Does the name resolve to an address? Does the address answer at the IP level? Is the port open, and does the service answer on it? Does the application return a correct answer? Each question is answered by a different tool, and each answer narrows the suspect until only one layer is left. The point of the ladder is that it eventually ends: a good debugging session shrinks the set of possible causes instead of growing it.

Each tool proves something narrow, and using it for more than that is how people reach wrong conclusions. ping proves that a host answers ICMP and that a path exists in both directions; it does not prove a web server is running. dig proves what a resolver answered and how long the answer is cached; it does not prove that the address is reachable. curl proves what the whole stack does for one concrete request; it does not prove that other clients get the same answer. ss proves what is actually listening on this machine and on which address; it does not prove that a firewall lets anyone in.

$ dig api.example.org +short
198.51.100.20

$ dig api.example.org +short
198.51.100.20

$ dig api.example.org @1.1.1.1 +short
203.0.113.7

$ dig api.example.org @8.8.8.8 +short
203.0.113.7Two resolvers, two different answers — a cache and a stale record, not a broken network.

$ curl -v https://api.example.org/health
*   Trying 198.51.100.20:443...
* connect to 198.51.100.20 port 443 failed: Connection timed out
* Failed to connect to api.example.org port 443: Connection timed out

$ curl -sv https://api.example.org/health -o /dev/null
*   Trying 93.184.216.34:443...
* Connected to api.example.org (93.184.216.34) port 443
* Server certificate: CN=*.example.org
< HTTP/2 200
< content-type: application/jsonThe same command with a timeout on the port and with the address that actually works.

ss deserves special attention because it separates two failures that look identical from outside: nothing is listening, or something is listening but only on the loopback address. A server bound to 127.0.0.1 accepts connections from the same machine and refuses every connection from outside, no matter what the firewall says. That single distinction explains a large share of the cases where the service works locally and is unreachable from anywhere else.

$ ss -tlnp
State   Local Address:Port   Process
LISTEN  127.0.0.1:8080       users:(("node",pid=4127,fd=21))
LISTEN  0.0.0.0:22           users:(("sshd",pid=880,fd=3))

$ curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:8080/
200One service is reachable from the network, the other only from this machine.