Reliability
SRE Fundamentals — SLIs, SLOs, error budgets, and burn-rate alerts
Define SLIs and SLOs, turn them into an error budget, and alert with multi-window multi-burn-rate rules that page on fast burn and ticket on slow burn — with the 99.9%/30-day math worked end to end.
Quick reference — reach for these first
SLI = good events / valid eventsWhat you measure — a ratio users actually feel (success, fast-enough).SLO: SLI >= 99.9% over 30dInternal target for the SLI. Set it achievable, then ratchet up.budget = 1 - SLO = 0.1%Allowed unreliability = 43.2 min / 30 days. Spend it deliberately.burn = (1 - SLI) / (1 - SLO)Multiples of budget spend; 1x = on pace to use it all exactly at period end.SLA = SLO + consequencesContract with penalties. Keep the SLO STRICTER so you act before it breaches.Multi-window multi-burn-rate: when to page vs ticket
Text equivalent: Error budget equals one minus the SLO. For a 99.9 percent SLO that is 0.1 percent, about 43.2 minutes over 30 days. Burn rate is the observed error ratio divided by the budget rate, and budget spent equals burn rate times window over period. A 14.4x burn over a 1-hour window confirmed by a 5-minute window spends 2 percent of the monthly budget and pages. A 6x burn over 6 hours confirmed by 30 minutes spends 5 percent and pages. A 1x burn over 3 days confirmed by 6 hours spends 10 percent and files a ticket. Page when the budget would be gone in hours; ticket when the burn is real but slow.
Common tasks
Step · PromQL / formula · Done when
| Step | PromQL / formula | Done when |
|---|---|---|
| Pick one user-journey SLI (availability or latency) | SLI = good events / valid events | The SLI tracks something users feel, not an internal counter |
| Define 'good' events as a query | GOOD = REQUESTS_TOTAL{job="SERVICE",code!~"5.."} | Query returns the count of successful requests |
| Record the SLI ratio over the window | job:sli:ratio30d = sum(rate(GOOD[30d])) / sum(rate(VALID[30d])) | Recorded value sits between 0 and 1 |
| Set the SLO target and window | SLO = 99.9% over 30d | Target is at or below what the last 28 days actually achieved |
| Compute the error budget | (1 - 0.999) * 30*24*60 | = 43.2 minutes of allowed badness per month |
| Add fast-burn PAGE alert [prod] | ratio1h > 14.4*0.001 AND ratio5m > 14.4*0.001 | Fires only when ~2% of the budget burns in 1h |
| Add mid-burn PAGE alert [prod] | ratio6h > 6*0.001 AND ratio30m > 6*0.001 | Fires when ~5% of the budget burns in 6h |
| Add slow-burn TICKET alert [prod] | ratio3d > 1*0.001 AND ratio6h > 1*0.001 | Fires when ~10% of the budget burns over 3 days |
| Dashboard the budget remaining | 1 - (1 - ratio30d)/0.001 | Panel shows percent of monthly budget still left |
| Route alerts: page vs ticket [prod] | severity=page -> PAGER ; severity=ticket -> QUEUE | Page wakes on-call; ticket lands in the work queue, not the pager |
| Assign incident roles when a page fires | IC + COMMS + OPS | Exactly one Incident Commander owns coordination |
| Write the postmortem trigger threshold | user-visible outage > THRESHOLD OR data loss | The trigger is documented, not decided case-by-case |
| Enforce the budget policy on exhaustion [CHANGES PROD] | budget <= 0 -> freeze feature launches | Freeze is pre-agreed by dev and SRE, not negotiated mid-incident |
| Review the SLO every quarter | compare SLO vs achieved SLI vs complaints | SLO still tracks user happiness; adjust if it drifted |
99.9% / 30-day SLO: budget, burn rate, and the page-vs-ticket call
page-vs-ticket
30 * 24 * 60
# 43200 minutes in the 30-day window(1 - 0.999) * 43200
# 43.2 minutes of allowed downtime per month0.0144 / 0.001
# burn rate = 14.4x14.4 * (60 / 43200)
# 0.02 = 2% of the WHOLE monthly budget in one hour43200 / 14.4
# 3000 min = 50h -> fast, severe -> PAGE (gated by a 5m window)0.001 / 0.001
# burn rate = 1x -> 10% of budget over 3 days -> TICKET, not a pageTroubleshooting — symptom → first check → next check
| Symptom | First check | Establishes | Next check |
|---|---|---|---|
| SLO dashboard is green but users are complaining | Is the SLI measured at the user's edge (LB/client), not server-internal? | The SLI is a proxy, not the real experience | Move measurement to the edge and tighten the SLO so green means happy |
| Budget is always exhausted; on-call paged constantly | Is the SLO target above what the service hit in the last 28 days? | The SLO is stricter than reality (too strict) | Loosen the SLO to an achievable number, then ratchet it up as you improve |
| Pages fire but almost no budget was actually spent | Does the alert require BOTH a long and a short window to be burning? | Missing short-window gate makes the alert flap | Add the short-window condition (5m with 1h) so brief blips self-resolve |
| Budget was blown before anyone got paged | Is there a fast-burn 14.4x/1h alert, not only 24h/3d rules? | Only slow-burn alerting is configured (too late) | Add the 1h/5m 14.4x page so severe burns alert within minutes |
| Error budget has hit zero | Is the feature-freeze policy written down and pre-agreed? | No budget policy to invoke | Freeze risky launches and spend the cycle on reliability until budget recovers |
| Team can't agree whether an incident breached the target | Is there one documented SLI definition (good vs valid events)? | No agreed SLI exists | Write one SLI spec per journey before setting or debating any SLO |
| Alert thresholds are set to the SLA number (e.g. 99.5%) | Are you alerting on the SLA or on a stricter internal SLO? | SLA/SLO confusion — alerting at the contract line | Set the internal SLO tighter than the SLA so you act before the contract breaches |
References
- Google SRE Book — Service Level Objectivessre.google
- Google SRE Workbook — Implementing SLOssre.google
- Google SRE Workbook — Alerting on SLOssre.google
- Google SRE Book — Managing Incidentssre.google
- Google SRE Book — Postmortem Culturesre.google
Math uses a 30-day rolling window (43,200 min) and event-based SLIs. Burn-rate windows and thresholds follow the Google SRE Workbook (14.4× / 6× / 1×). Swap CAPS PLACEHOLDERS (REQUESTS_TOTAL, SERVICE, THRESHOLD) for your own metric and label names.