← All cheatsheets

AI CLI

AI Output Evaluation — know if the model is actually right

Build a golden set, score with exact-match + rubrics + LLM-as-judge, track accuracy/precision/recall alongside cost and latency, and gate every prompt or model change on a regression run so silent quality drops get caught before users do.

Docs-checkedevaluationevalsllm-judgegolden-setmetrics

Quick reference — reach for these first

1. Golden setlabeled.jsonl # {input, expected} pairsFreeze a representative, labeled set BEFORE tuning. This is your source of truth.
2. Exact-matchassert out.strip() == expectedCheapest, deterministic check for structured/classification outputs. Run it first.
3. Rubric scorescore = sum(criteria_met)/len(criteria)Deterministic 0-1 per-criterion scoring for open-ended text (has citation? on-topic? correct format?).
4. LLM-as-judgejudge(prompt, output) -> {score, reason}Grade subjective quality with a model — but validate the judge against human labels first.
5. Regression runeval baseline vs candidate; diff metricsRe-run the full suite on every prompt/model change. Block the merge on any drop.

The evaluation loop: labeled set in, a gated ship/block decision out

AI output evaluation loopGolden set feeds a model run; outputs are scored by exact-match, rubric, and a calibrated LLM-judge; metrics plus cost and latency reach a regression gate that ships or blocks; failures feed back into the golden set.Golden setlabeled {in,expected}Runprompt + modelExact-matchRubric scoreLLM-judge (calibrated)Metricsacc / P / R / passRegression gateship if up + cost flat; else block+ cost & latencyfeedback: escaped prod failures become new labeled cases

Text equivalent: A left-to-right pipeline. A frozen golden set of labeled examples feeds a run of the prompt and model. Outputs are scored three ways — exact-match, rubric, and a calibrated LLM-as-judge. Scores roll up into metrics: accuracy, precision, recall, pass-rate, plus cost and latency. A regression gate compares candidate against baseline and either ships the change or blocks it, and blocked or escaped failures are fed back into the golden set.

Common tasks

Build the set · Score it · Gate changes

Build the setScore itGate changes
Define success criteria and the metric that maps to it# write down: task, target metric (accuracy? pass-rate?), thresholdOne sentence states what 'good' means and the single number you will gate on.
Collect 50-200 real, representative examplessample from prod logs + known edge/failure casesClass balance and input distribution roughly match production traffic.
Label expected outputs (human or trusted source)two labelers; reconcile disagreementsInter-labeler agreement is high; disagreements are documented, not guessed.
Pin determinism for reproducible runstemperature=0, fixed seed, pinned model idSame input twice gives the same output and the same score.
Implement exact-match / structural scoringnormalize() then ==; validate JSON schemaScorer passes on a hand-checked correct output and fails on a wrong one.
Write a rubric for open-ended outputslist binary criteria -> fraction met = scoreTwo people scoring the same output with the rubric get the same number.
Build an LLM-as-judge for subjective qualityjudge with a rubric prompt; return score + reasonJudge outputs parse cleanly and include a justification for each score.
Calibrate the judge against human labelsjudge the labeled set; measure agreement vs humansJudge/human agreement is high (e.g. >0.8) BEFORE you trust it unsupervised.
Compute aggregate + per-class metricsaccuracy, precision, recall, pass-rate per classYou see per-class numbers, not just one average that hides a weak class.
Record cost and latency per runsum input+output tokens * price; p50/p95 latencyQuality, $/1k calls, and p95 latency all appear in one results table.
Snapshot a baseline resulteval run --save baseline.jsonbaseline.json holds the current prompt/model scores for later diffing.
Regression-test every change against baselineeval run candidate; diff vs baseline # gate in CIMerge is blocked automatically if any gated metric regresses.
Feed real prod failures back into the setadd escaped bugs as new labeled casesEach incident becomes a permanent regression case so it cannot silently return.

Evaluate a 3-class sentiment classifier prompt and gate a prompt change

classification eval

1
Freeze a small labeled golden set (SAMPLE INPUT). Each line is an input and its human-verified expected class.
# labeled.jsonl
{"text":"Shipping was fast and the case feels premium.","expected":"positive"}
{"text":"It works but the app is clunky and slow.","expected":"neutral"}
{"text":"Battery died in two days. Returning it.","expected":"negative"}
{"text":"Exactly what I needed, five stars.","expected":"positive"}
{"text":"Arrived on time. Nothing special.","expected":"neutral"}
{"text":"Terrible support, never replied.","expected":"negative"}
{"text":"Love it, would buy again.","expected":"positive"}
{"text":"Does the job, a bit overpriced.","expected":"neutral"}
# 8 labeled cases, balanced across the 3 classes
2
Pin the copy-ready classification PROMPT and force deterministic, parseable output.
SYSTEM = (
  "Classify the sentiment of the user text as exactly one word: "
  "positive, neutral, or negative. Output only that word, lowercase."
)
# call with temperature=0, pinned model id (e.g. a current Claude Sonnet),
# so runs are reproducible
# model returns a single token like: positive
3
Score with exact-match and compute accuracy + per-class precision/recall (the SCORING CODE).
import json, collections
rows=[json.loads(l) for l in open("labeled.jsonl")]
cls=["positive","neutral","negative"]
tp=collections.Counter(); fp=collections.Counter(); fn=collections.Counter()
correct=0
for r in rows:
    pred=classify(r["text"]).strip().lower()  # your model call
    exp=r["expected"]
    if pred==exp: correct+=1; tp[exp]+=1
    else: fp[pred]+=1; fn[exp]+=1
acc=correct/len(rows)
print(f"accuracy={acc:.2f} ({correct}/{len(rows)})")
for c in cls:
    p=tp[c]/(tp[c]+fp[c]) if tp[c]+fp[c] else 0
    rec=tp[c]/(tp[c]+fn[c]) if tp[c]+fn[c] else 0
    print(f"{c:8} precision={p:.2f} recall={rec:.2f}")
# deterministic scorer; no LLM-judge needed for closed-set labels
4
Read the EXPECTED METRIC OUTPUT of the baseline run.
python eval.py
# accuracy=0.88 (7/8)
# positive  precision=1.00 recall=1.00
# neutral   precision=0.67 recall=0.67
# negative  precision=1.00 recall=1.00
# tokens/call~45  p95 latency~0.6s  cost~flat
5
EVALUATE the change: run the candidate prompt, diff against baseline, and decide ship vs block.
python eval.py --prompt candidate > cand.txt
diff baseline.txt cand.txt
# candidate: accuracy=0.88->0.94, neutral recall 0.67->0.83, cost/latency unchanged
# DECISION: ship — pass-rate up, no class regressed, cost flat.
# If any class recall had dropped or cost spiked: BLOCK the change.

Troubleshooting — symptom → first check → next check

SymptomFirst checkEstablishesNext check
Eval set is tiny or unrepresentative (passes everything)Compare set's class balance and input length distribution to a week of prod logs.Whether the set exercises the cases users actually send, not just easy ones.Grow to 50-200 examples, oversample known edge/failure cases, rebalance classes.
LLM-judge scores disagree with human reviewersRun the judge over your human-labeled set and compute agreement rate.Whether the judge is trustworthy or just confidently wrong.Tighten the rubric, add few-shot graded examples, re-calibrate until agreement is high; never ship an unvalidated judge.
Metric looks great but users complainTrace 5 complaints to their eval cases — are they even in the set, and does the metric reward them?Whether you are measuring the wrong thing (e.g. accuracy when task-success is what matters).Switch to a metric tied to the real task (task-success, recall on the costly class) and add the complaint cases.
Eval scores flap between identical runsCheck temperature, seed, and whether the model id is pinned (not an auto-updating alias).Whether nondeterminism is in sampling vs the scorer itself.Set temperature=0, pin seed and exact model id; make the scorer deterministic; median over N runs if still noisy.
Eval passes but production fails (distribution shift)Diff recent prod inputs against the eval set — new topics, formats, or languages?That prod has drifted away from what the frozen set covers.Continuously sample fresh prod traffic into the set and re-label; add every escaped failure as a case.
A change ships, aggregate score flat, one class collapsesOpen the per-class precision/recall table, not the single average.Whether an average is masking a regression in a minority class.Gate on per-class floors too, not just the overall number; block the change until the class recovers.
Quality improved but cost/latency explodedCompare tokens-per-call and p95 latency in the candidate vs baseline row.Whether the gain is worth the spend — a bigger model or longer prompt may not pay off.Report quality AND cost AND latency together; decide against the budget, or try a smaller model / shorter prompt.

References

Scoring examples use Python with a closed-label set, so exact-match suffices and no judge is needed; model calls run at temperature=0 with a pinned current Claude Sonnet/Opus 4.x id for reproducibility. Model ids and pricing are referenced generically — confirm the exact id and token prices in the Anthropic docs before wiring cost math. The LLM-as-judge pattern is only for open-ended/subjective outputs and must be calibrated against human labels before it gates anything.