← All cheatsheets

Network

Deep network troubleshooting — trace a request client to backend

Follow one failing request across DNS, routing, TCP, TLS, HTTP and app health — and read the distinct error each layer produces so you fix the right one.

Docs-checkeddnstcptlshttpmturouting

Quick reference — reach for these first

dnsdig +short app.shipreadymetrics.com AThe resolver's answer only. Empty = NXDOMAIN or no record. Compare resolvers with dig @RESOLVER.
pathmtr -rwzbc 50 scm.ccPer-hop loss and latency (needs install + root). Loss that starts mid-path AND persists to the end is a real drop.
tcpnc -vz scm.cc 443Reachability of one port. 'succeeded' vs 'refused' (RST, port closed) vs timeout (filtered path).
tlsopenssl s_client -connect scm.cc:443 -servername scm.ccFull handshake: cert chain, expiry, SNI, negotiated protocol. Append </dev/null to return.
timingcurl -sS -o /dev/null -w '%{time_namelookup} %{time_connect} %{time_appconnect} %{time_starttransfer} %{time_total}\n' https://scm.ccSplits the wall-clock into DNS / TCP / TLS / TTFB so you know which stage is slow.

The request path and the error each stage produces

The request path from client to backend and the error each stage produces A request crosses six stages: DNS resolution, routing and interface, TCP connect, TLS handshake, HTTP load balancer, and app health. DNS failure shows NXDOMAIN or SERVFAIL. Routing shows no route or unreachable. TCP shows connection refused, which is an RST meaning the port is closed, or a timeout meaning the path is filtered. TLS shows a certificate, SNI, or protocol error. The load balancer and app show 502 bad gateway or 504 gateway timeout. client backend 1 · DNSresolve 2 · routeiface 3 · TCPconnect 4 · TLShandshake 5 · HTTPload bal. 6 · apphealth NXDOMAINSERVFAIL no route /unreachable refused = RSTtimeout = filtered cert / SNI /proto error 502 badgateway 504 / 5xxtimeout Each layer fails its own way — match the symptom to the stage before you touch the app.

Text equivalent: A request crosses six stages left to right: 1 DNS resolve, 2 route/interface, 3 TCP connect, 4 TLS handshake, 5 HTTP load balancer, 6 app health. Each stage fails in its own recognizable way. DNS: NXDOMAIN or SERVFAIL. Routing: no route or unreachable. TCP: 'connection refused' is an RST meaning the port is closed, while a timeout means the path is filtered. TLS: a certificate, SNI, or protocol error. The load balancer returns 502 bad gateway when the upstream is unreachable; the app layer returns 504 or other 5xx when the backend is too slow or hung. Match the symptom to the stage before touching the app.

Common tasks

Task · Command · What to check

TaskCommandWhat to check
Resolve a namedig +short app.shipreadymetrics.comReturns an A/AAAA address? Empty answer = NXDOMAIN or no such record.
Compare resolversdig @1.1.1.1 scm.cc +short; dig @8.8.8.8 scm.cc +shortDifferent answers = stale cache or split-horizon DNS, not the app.
Trace the delegationdig +trace scm.ccWhich nameserver fails; distinguishes SERVFAIL (broken zone) from NXDOMAIN (no name).
Show the route to a hostip route get 203.0.113.10Which interface and gateway the kernel picks; 'unreachable' = no route.
Per-hop loss and latencymtr -rwzbc 50 scm.ccInstall mtr; needs root. Loss appearing mid-path and persisting to the last hop is real.
Test a single TCP portnc -vz scm.cc 443'succeeded' = open; 'refused' = RST (closed); hang/timeout = filtered.
Watch retransmits livess -tin 'dst scm.cc'-i shows retrans, rtt, cwnd. Climbing retrans = packet loss on the path.
Probe the path MTUping -M do -s 1472 app.shipreadymetrics.com1472+28=1500. Fails with NO 'frag needed' reply = MTU black hole. macOS: ping -D -s.
IPv4 vs IPv6curl -4 https://scm.cc; curl -6 https://scm.ccOne works, the other hangs = a broken AAAA / IPv6 path. Check records and routing.
Capture the handshakesudo tcpdump -ni any host scm.cc and port 443SYN [S] going out with no SYN-ACK [S.] back = dropped/filtered, not the app.
Inspect the TLS sessionopenssl s_client -connect scm.cc:443 -servername scm.cc </dev/nullVerify-return-code 0, cert not expired, SNI served, protocol matches.
Time every phasecurl -w '@curl-format.txt' -o /dev/null -sS https://scm.ccDNS vs connect vs appconnect(TLS) vs starttransfer(TTFB) isolates the stage.
Check the backend directlycurl -sS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/healthz502 at the LB but 200 here = the LB→app hop, not the app itself.

HTTPS works locally but times out from another network

INCIDENT A

1
Reproduce from the affected network with a hard timeout so it fails fast.
curl -sS -m 10 https://app.shipreadymetrics.com/healthz
# curl: (28) Connection timed out after 10001 ms
2
Rule out DNS first — does the name resolve, and to the same IP as the network where it works?
dig +short app.shipreadymetrics.com
# 203.0.113.10   (identical to the working network → not DNS)
3
Refused or filtered? A timeout, not an RST, means the packet is being dropped — a firewall, not a dead port.
nc -vz 203.0.113.10 443
# nc: connect to 203.0.113.10 port 443 (tcp) timed out: Operation now in progress
4
Confirm the SYN leaves and nothing comes back.
sudo tcpdump -ni any host 203.0.113.10 and port 443
# IP ... > 203.0.113.10.443: Flags [S] ...   (repeats, no [S.] reply)
!
From the working network the same nc returns 'succeeded' and the port is open. DNS is identical and the service is up — so the path from THIS network is filtered, not broken DNS and not a down app.
✓
Allowlist the source range on the firewall / security group (client-side egress or server-side ingress). Re-run nc until it reads 'succeeded' and curl returns 200 — that confirms the fix was the path, not the app.

Small requests succeed but large transfers stall (path-MTU black hole)

INCIDENT B

1
A small health check is instant; a large download hangs then dies — after the handshake, mid-transfer.
curl -sS -o /dev/null -w '%{http_code} %{size_download}\n' https://app.shipreadymetrics.com/report.csv
# (hangs ~20s) curl: (28) Operation too slow ... 0 bytes received
2
TLS clearly completed (small packets got through), so suspect MTU, not TLS. Confirm appconnect is fine and the stall is in the transfer.
curl -sS -o /dev/null -w '%{time_appconnect} %{time_starttransfer}\n' https://scm.cc/report.csv
# 0.11  (appconnect OK; then stalls before first byte of bulk data)
3
Probe path MTU with the DF bit set and sweep the size. A full-size packet that fails with NO 'frag needed' reply is a black hole (the ICMP is filtered). 1472 payload + 28 = 1500.
ping -M do -s 1472 app.shipreadymetrics.com   # then -s 1473
# 1472: 64 bytes ... ok   ·   1473: (no reply at all — not even 'frag needed')
4
Find the ceiling that does get through — it reveals the real path MTU (here a tunnel/VPN overhead of ~100 bytes).
ping -M do -s 1372 app.shipreadymetrics.com
# ok → path MTU ≈ 1400
✓
Clamp MSS or lower the interface MTU to the discovered path (e.g. ip link set dev DEV mtu 1400, or MSS clamp on the gateway). Re-run the large curl → 200 with the full body. On macOS/BSD the probe is ping -D -s 1472.

Troubleshooting — symptom → first check → next check

SymptomFirst checkEstablishesNext check
NXDOMAIN / SERVFAILdig +short HOST; dig +trace HOSTName doesn't exist vs resolver/zone brokenConfirm the record exists; compare @1.1.1.1 against the internal resolver
"Connection refused"nc -vz HOST PORT (gets an RST)Host reachable, but nothing listening on that portConfirm the service is up and bound to the right address:port
"Connection timed out"nc -vz HOST PORT hangs; tcpdump shows SYN, no SYN-ACKPacket filtered/dropped on the pathFirewall / security group / routing — not the app
TLS handshake erroropenssl s_client -connect HOST:443 -servername HOSTExpiry / chain / SNI / protocol mismatchFix cert or SNI; test the intended host with curl --resolve
502 Bad Gatewaycurl -s -o /dev/null -w '%{http_code}' to the backend directlyLB is up but the upstream is unreachable/crashedCheck the app process and the LB→app port/health check
504 Gateway Timeoutcurl -w '%{time_total}' against the backendUpstream reachable but too slow / hungBackend latency, connection-pool limits, proxy read timeouts
Large transfers stall, small ones fineping -M do -s 1472 HOST, then sweep upwardPath-MTU black hole (frag-needed ICMP filtered)Clamp MSS or lower the interface MTU to the real path MTU
Every stage passescurl -w full timing to the real endpointRequest path is healthy end-to-endSave the timing as a baseline to diff against next incident

References

Linux with iproute2 (ip, ss) and GNU/iputils ping, curl ≥ 7.75, OpenSSL ≥ 3. mtr and traceroute usually need installing and raw-socket privileges (root/suid). macOS/BSD differ: ping uses -D (not -M do) to set the DF bit, and ss is replaced by netstat/lsof. app.shipreadymetrics.com and scm.cc are example hosts; replace them. Placeholders in CAPS.