System
System Design 101 — load balancing, caching, queues, and the trade-offs
The five building blocks of a scalable backend — load balancing, caching, databases, queues, and replication — and the consistency, latency, availability, and cost trade-offs at every hop of a request.
Quick reference — reach for these first
nginx / HAProxy / ALBSpreads traffic across N app instances and health-checks them, routing around dead nodes. Trades a hop of latency for availability + horizontal scale.Redis / MemcachedKeeps hot data in memory near the app (cache-aside + TTL). Cuts read latency and DB cost, at the risk of serving stale data.Postgres / DynamoDBDurable system of record. SQL gives joins + ACID; NoSQL gives horizontal scale + flexible schema. Index the columns you filter on.Kafka / SQS / RabbitMQDecouples producers from consumers. Absorbs spikes and moves slow work off the request path, trading immediate consistency for throughput.leader + read replicasFollowers add read capacity and availability; writes still go to the single leader. Replica lag means reads are eventually consistent.One request: client to load balancer to cache/app to database, with a queue for async work
Text equivalent: Left to right the flow is Client, Load Balancer, App, Database. The App also talks to a Cache drawn above it (cache-aside with a TTL, a two-way arrow) and pushes async work down into a Queue, which a Worker beside the Database consumes.
Common tasks
Task · Command · What to check
| Task | Command | What to check |
|---|---|---|
| Put a load balancer in front of the app tier | upstream app { server APP1:PORT; server APP2:PORT; } | curl -s http://LB_HOST/whoami returns different backend IDs across calls |
| Add health checks so the LB drops dead nodes | check: GET /health interval=INTERVAL fails=N | Stop APP1; traffic shifts entirely to APP2 within INTERVAL*N |
| Add a cache-aside read path in front of the DB | GET KEY -> miss -> SELECT ... -> SET KEY VALUE EX TTL | Second read of KEY is served from cache: redis-cli GET KEY returns a value |
| Put a TTL on every cache key so staleness is bounded | redis-cli SET KEY VALUE EX SECONDS | redis-cli TTL KEY returns a positive, shrinking number |
| [PROD] Invalidate the cache on write to avoid stale reads | UPDATE TABLE ...; redis-cli DEL KEY | Read immediately after the write returns the new value, not the old one |
| Index the columns your hot queries filter on | CREATE INDEX idx_col ON TABLE(COLUMN) | EXPLAIN on the query shows an Index Scan, not a Seq Scan |
| Add a read replica and route reads to it | writes -> LEADER ; SELECT -> REPLICA | Replica lag stays under THRESHOLD: SHOW REPLICA STATUS / pg_stat_replication |
| Move slow side-effects off the request path | producer -> QUEUE -> worker consumes async | API returns 2xx in ms; the job completes later and its effect appears |
| Add a dead-letter queue for poison messages | maxReceiveCount=N -> route to DLQ | A message that always fails lands in the DLQ instead of retrying forever |
| Add jitter + backoff to prevent a cache stampede | ttl = BASE + random(0, JITTER) ; retry sleep = 2^n + rand() | A mass TTL expiry no longer spikes DB connections to the ceiling |
| [PROD] Eliminate single points of failure across AZs | run 2+ of every tier across AZ_A, AZ_B behind the LB | Drain AZ_A; the service stays up and serves requests from AZ_B |
| Load-test to find the real bottleneck before scaling | k6 run --vus VUS --duration DURATION script.js | p99 latency stays under SLO at target RPS; the saturating tier is identified |
Trace one read, then one write, through LB -> cache -> app -> DB (+ queue)
read + write path
curl -s https://APP_HOST/users/USER_ID
# routed to app-node-2 (app-node-1 draining)redis-cli GET user:USER_ID
# (hit) {"name":"Ada",...} TTL 42SELECT * FROM users WHERE id = USER_ID; -- on REPLICA
# 1 row (replica ~200 ms behind leader)redis-cli SET user:USER_ID '{...}' EX 60
# OKUPDATE users SET name='Ada L.' WHERE id=USER_ID; -- on LEADER
# UPDATE 1redis-cli DEL user:USER_ID
# (integer) 1PUBLISH jobs '{"type":"reindex","id":USER_ID}'
# queued; API returns 200 in ~5 msSELECT name FROM users WHERE id=USER_ID; -- LEADER
# Ada L.Troubleshooting — symptom → first check → next check
| Symptom | First check | Establishes | Next check |
|---|---|---|---|
| DB CPU spikes every time a popular cache key expires | Correlate DB load spikes with the TTL expiry of HOT_KEY (cache logs + DB metrics) | Thundering herd: many requests miss at the same instant and all stampede the DB | Add request coalescing / single-flight so one miss repopulates, stagger TTLs with jitter, or refresh-ahead before expiry |
| One cache or DB node runs hot while the others sit idle | Measure per-key / per-partition request rate and find the skewed KEY or shard | Hot key or hot partition: traffic is concentrated on a single node, not spread | Add a small in-process cache for the key, replicate it to several nodes, or shard with a better key to spread load |
| Users see stale data immediately after they save a change | Measure replica lag (seconds behind leader) and confirm whether reads are routed to a REPLICA | Replica lag + eventual consistency: the read hit a follower that had not yet applied the write | Route read-after-write to the LEADER, or pin a user's session to read-your-writes until lag clears |
| Queue depth climbs steadily and never drains | Compare producer rate vs. consumer throughput and check worker error / DLQ counts | Backlog: consumers cannot keep up — under-provisioned, stuck, or crash-looping on bad messages | Scale out consumers, fix or DLQ poison messages, and autoscale workers on queue depth |
| The whole service goes down when a single node dies | Map every tier and find any component running with only one instance or one leader | Single point of failure: that tier has no redundancy or failover | Run 2+ instances per tier across AZs behind the LB; add automated leader failover for the database |
References
- AWS Well-Architected Framework — Reliability Pillardocs.aws.amazon.com
- AWS — What is Elastic Load Balancing?docs.aws.amazon.com
- Redis — EXPIRE / key expiration (TTL)redis.io
- PostgreSQL — Indexeswww.postgresql.org
- Apache Kafka — Documentation (Design)kafka.apache.org
Examples use nginx, Redis, PostgreSQL, and a generic message queue. The concepts map to any equivalent stack: HAProxy or an ALB for load balancing, Memcached for caching, MySQL or DynamoDB for storage, and Kafka, SQS, or RabbitMQ for queues. CAP-style trade-offs are framed as consistency vs. latency vs. availability vs. cost, not a single "right" answer.