← All cheatsheets

Reliability

GCP SRE — the first 15 minutes of an incident

A first-15-minutes triage runbook for Google Cloud incidents: confirm which account, project and region you are in, establish blast radius, read the four golden signals, check load-balancer backend health, correlate recent deploys/config/IAM changes from Audit Logs, then decide mitigate vs roll back against the affected SLI. Inspect/read steps first; rollbacks and prod-changing actions are labeled.

Docs-checkedgcpsreincidentmonitoringlogging

Quick reference — reach for these first

Where am I?gcloud config listConfirm active account, PROJECT_ID and REGION before touching anything
Blast radiusgcloud logging read 'resource.type="http_load_balancer" AND httpRequest.status>=500' --freshness=15m --project=PROJECT_IDCount 5xx and see which services/regions are hit — read-only
Golden signalsgcloud monitoring dashboards list --project=PROJECT_IDLatency, traffic, errors, saturation in Metrics Explorer
Backend healthgcloud compute backend-services get-health BACKEND_SERVICE --global --project=PROJECT_IDAre LB backends HEALTHY, or has capacity gone?
What changed?gcloud logging read 'logName="projects/PROJECT_ID/logs/cloudaudit.googleapis.com%2Factivity"' --freshness=30m --project=PROJECT_IDRecent deploys, config and IAM edits from Audit Logs

First 15 minutes — GCP incident triage

First 15 minutes of a GCP incidentSix-step triage flow: confirm account, project and region; establish blast radius; read the four golden signals latency, traffic, errors and saturation; check load-balancer and backend health; correlate recent deploy, config and IAM changes; then decide mitigate or roll back against the affected SLI.Incident triage — inspect first, change last1 Confirmacct · project · region2 Blast radiuswho · what · where3 Goldensignals4 LB / backendhealth5 Recent changesdeploy · config · IAM6 Mitigate /roll back vs SLIlatency · traffic · errors · saturation

Text equivalent: A six-step left-to-right flow for the first fifteen minutes of a Google Cloud incident. Step 1: confirm account, project and region. Step 2: establish blast radius — who, what and where is affected. Step 3: inspect the four golden signals, shown below as latency, traffic, errors and saturation. Step 4: check load-balancer and backend health. Step 5: correlate recent changes across deploy, config and IAM from Audit Logs. Step 6, highlighted as the decision: mitigate or roll back, measured against the affected SLI.

Common tasks

Triage step · Command · What it tells you

Triage stepCommandWhat it tells you
Confirm active account, project and regiongcloud config list && gcloud auth listaccount / project / region match the affected environment, not a stale default
Establish blast radius: count 5xx in last 15mgcloud logging read 'resource.type="http_load_balancer" AND httpRequest.status>=500' --freshness=15m --project=PROJECT_ID --format='value(timestamp)' | wc -lerror count vs normal baseline tells you severity and whether it is growing
Scope which service/region is affectedgcloud logging read 'resource.type="http_load_balancer" AND httpRequest.status>=500' --freshness=15m --project=PROJECT_ID --format='value(resource.labels.backend_target_name)' | sort | uniq -cone backend/region concentrated = localized; everything = platform-wide
Traffic signal: is request volume normal?gcloud monitoring dashboards list --project=PROJECT_ID # then Metrics Explorer: loadbalancing.googleapis.com/https/request_counta sudden drop = upstream outage; a spike = load/DDoS driving saturation
Errors signal: split edge vs backend 5xxgcloud logging read 'resource.type="http_load_balancer"' --freshness=15m --project=PROJECT_ID --format='value(jsonPayload.statusDetails)' | sort | uniq -cstatusDetails names the layer (LB vs backend_connection_* vs app)
Latency signal: p95/p99 from LB logsgcloud logging read 'resource.type="http_load_balancer"' --freshness=15m --project=PROJECT_ID --format='value(httpRequest.latency)' | sort -rn | headtail latency spike points at a slow backend or dependency
Saturation signal: CPU/mem/instanceskubectl top pods -n NAMESPACE # GKE — or Cloud Run: gcloud run services describe SERVICE --region=REGION --project=PROJECT_ID --format='value(status.traffic)'pegged CPU/mem or maxed instance count = capacity-bound, needs scale or shed
Load-balancer backend healthgcloud compute backend-services get-health BACKEND_SERVICE --global --project=PROJECT_IDall HEALTHY rules out the LB/health-check layer; UNHEALTHY points there
Target/app health (GKE or Cloud Run)kubectl get pods -A --field-selector=status.phase!=Running # GKE — or: gcloud run revisions list --service=SERVICE --region=REGION --project=PROJECT_IDcrashlooping pods / failing revision correlates the app layer to the errors
VPC / connectivity sanity checkgcloud compute firewall-rules list --project=PROJECT_ID --filter='disabled=false' --format='table(name,direction,sourceRanges.list(),allowed[].map().firewall_rule().list())'no recently added deny / removed allow rule blocking backend or health-check traffic
Correlate recent changes (Admin Activity audit log)gcloud logging read 'logName="projects/PROJECT_ID/logs/cloudaudit.googleapis.com%2Factivity"' --freshness=30m --project=PROJECT_ID --limit=20 --format='table(timestamp,protoPayload.methodName,protoPayload.authenticationInfo.principalEmail)'a deploy/config change whose timestamp lines up with error onset is your prime suspect
Check recent IAM changesgcloud logging read 'protoPayload.methodName="SetIamPolicy"' --freshness=6h --project=PROJECT_ID --limit=20 --format='table(timestamp,protoPayload.authenticationInfo.principalEmail,resource.type)'a removed binding explains sudden 403/PERMISSION_DENIED between services
Inventory recent deploys / last-good versiongcloud run revisions list --service=SERVICE --region=REGION --project=PROJECT_ID --format='table(metadata.name,status.conditions[0].status,metadata.creationTimestamp)' # GKE: kubectl rollout history deployment/DEPLOYMENTyou can name the previous known-good revision to roll back to
Decide + act against the SLI (LABEL: prod change / rollback)# Cloud Run rollback — PROD CHANGE: gcloud run services update-traffic SERVICE --to-revisions=REVISION=100 --region=REGION --project=PROJECT_ID # GKE: kubectl rollout undo deployment/DEPLOYMENTerror-budget burn exceeds threshold -> roll back; marginal -> mitigate (scale/flag) and keep investigating

HTTP 5xx errors spike after a deployment

5xx after deploy

1
Confirm you are in the affected project and region
gcloud config list
# core.project = PROJECT_ID, compute/region = REGION, account = oncall@
2
Measure blast radius: 5xx count over the last 15 minutes
gcloud logging read 'resource.type="http_load_balancer" AND httpRequest.status>=500' --freshness=15m --project=PROJECT_ID --format='value(timestamp)' | wc -l
# 4123  — roughly 18x the normal baseline
3
Split edge vs backend: group the failures by statusDetails
gcloud logging read 'resource.type="http_load_balancer" AND httpRequest.status>=500' --freshness=15m --project=PROJECT_ID --format='value(jsonPayload.statusDetails)' | sort | uniq -c
# 4012 backend_connection_closed_before_data_sent_to_client -> backend, not edge
4
Check load-balancer backend health
gcloud compute backend-services get-health BACKEND_SERVICE --global --project=PROJECT_ID
# healthState: UNHEALTHY on 6/8 backends
5
Read app logs on the newest revision
gcloud run revisions list --service=SERVICE --region=REGION --project=PROJECT_ID
# newest REVISION rev-00042 (01:14Z) is serving 100% traffic and throwing 500s
!
Correlate recent changes in the Admin Activity audit log
gcloud logging read 'logName="projects/PROJECT_ID/logs/cloudaudit.googleapis.com%2Factivity"' --freshness=30m --project=PROJECT_ID --format='table(timestamp,protoPayload.methodName,protoPayload.authenticationInfo.principalEmail)'
# 01:14Z  google.cloud.run.v1.Services.ReplaceService  deployer@  -> rev-00042
7
Decide against the SLI: error-budget burn is critical, so roll back
# 18x baseline, fast burn on the availability SLO -> mitigate by reverting to last-good
# last-good = rev-00041 (healthy for 3 days)
✓
LABEL prod change — shift all traffic back to the last-good revision
gcloud run services update-traffic SERVICE --to-revisions=rev-00041=100 --region=REGION --project=PROJECT_ID
# Traffic: rev-00041 = 100%
✓
Verify recovery: re-run the blast-radius count and re-check backend health
gcloud logging read 'resource.type="http_load_balancer" AND httpRequest.status>=500' --freshness=2m --project=PROJECT_ID --format='value(timestamp)' | wc -l
# 3 — back to baseline; get-health now reports 8/8 HEALTHY

Troubleshooting — symptom → first check → next check

SymptomFirst checkEstablishesNext check
Clients see 504 / gateway timeoutgcloud compute backend-services get-health BACKEND_SERVICE --global --project=PROJECT_IDwhether backends are UNHEALTHY/at capacity vs the app itself being slowIf HEALTHY, read app latency logs and dependency calls; if UNHEALTHY, inspect health-check config and the last deploy, then roll back
DNS resolution fails / NXDOMAIN to a servicegcloud dns record-sets list --zone=ZONE --project=PROJECT_ID (and dig NAME)whether the record exists and has propagated vs a wrong/edited Cloud DNS zoneVerify record + TTL, then check Audit Logs for a recent DNS/config change and revert it
Callers suddenly get 403 / PERMISSION_DENIEDgcloud logging read 'protoPayload.methodName="SetIamPolicy"' --freshness=6h --project=PROJECT_IDwhether a recent IAM edit removed a required bindingDiff against gcloud projects get-iam-policy PROJECT_ID and restore the binding (LABEL: prod IAM change)
LB reports 0 healthy backends / 502sgcloud compute backend-services get-health BACKEND_SERVICE --global && gcloud compute health-checks describe HEALTH_CHECK --project=PROJECT_IDwhether the health check is misconfigured vs the instances/pods are actually downMatch HC port/path to the app; if a new revision broke it, roll back to the last-good version
429 / RESOURCE_EXHAUSTED, scale-up blockedgcloud logging read 'protoPayload.status.message:"Quota" OR severity>=ERROR' --freshness=15m --project=PROJECT_ID && gcloud compute regions describe REGION --project=PROJECT_IDwhich quota/capacity limit is being hit in this regionShed or rate-limit load, scale into another region, and request a quota increase
Error rate jumps right after a releasegcloud run revisions list --service=SERVICE --region=REGION (or kubectl rollout history deployment/DEPLOYMENT)whether the new revision's timestamp correlates with error onsetLABEL rollback: shift 100% traffic to the last-good revision (or kubectl rollout undo) and verify the SLI recovers

References

Assumes gcloud CLI 480+ (2025) on macOS/Linux bash/zsh, Google Cloud, kubectl for GKE; replace CAPS placeholders (PROJECT_ID, REGION, SERVICE, CLUSTER, BACKEND_SERVICE, DEPLOYMENT, REVISION) with your own.