← All cheatsheets

AI CLI

Agentic AI — building blocks, the loop, and safe operation

The core parts of an LLM agent — model, tools, state, and a human-approval gate — plus the perceive-plan-act-observe loop and how to test, monitor, and keep it safe.

Docs-checkedagentstoolsevaluationllmmonitoring

Quick reference — reach for these first

model + tool schema{"name":"search_docs","description":"Search our docs. Read-only.","input_schema":{"type":"object","properties":{"query":{"type":"string"}},"required":["query"]}}Give the model one typed contract per tool. A strict schema with required fields is what cuts hallucinated args.
agent loopperceive -> plan -> act -> observe -> repeatEach turn: read state, pick a tool, call it, feed the result back. Stop on done or MAX_STEPS.
approval gateif tool.side_effects: await human_approve(call)Read-only tools run free; any write/send/delete/pay waits for an explicit human yes.
validated callargs = Schema.validate(call.input) # reject, don't guessValidate tool input against the schema before running. On failure return an error the model can retry on.
eval metricsuccess = passed / total # + tool_error_rate, p95_latency, $ / taskScore the agent on a fixed question set before and after every change. 'Seems better' is not a number.

The agent loop — model, tools, state, and the approval gate

The agent loop with a human-approval gate on side-effecting tools The model repeats perceive, plan, act, observe. It reads from and writes to state/memory, calls read-only tools directly, and routes write, send, and delete tools through a human-approval gate before they run. Tool results return to the model as the observe step. loop until done or MAX steps State / memory history · scratchpad Model (LLM) Opus 4.x / Sonnet perceive plan · remember approve? human gate Read tools search · fetch (no gate) Write tools send · delete · pay act · read act · write approved observe · tool results return to the model

Text equivalent: An agent loop. The model (an LLM such as Opus 4.x or Sonnet) sits in the center. It perceives from State/memory and writes plans back to it (plan, remember). To act, it either calls Read tools (search, fetch) directly with no gate, or sends a write/send/delete/pay action through a human-approval gate (approve?) that only forwards the call once a human approves. Tool results flow back to the model as the observe step, and the whole cycle repeats until the task is done or a MAX-steps cap is hit.

Common tasks

Concept · Example / pattern · Why it matters

ConceptExample / patternWhy it matters
ModelOpus 4.x for planning; Sonnet for cheap high-volume stepsThe reasoning engine. Capability vs cost-per-step is your main tuning lever.
ToolsJSON-schema functions the model may callTools are the agent's only way to affect the world, so each is an error and attack surface.
State / memorymessages[] + a scratchpad or vector store of prior resultsWithout carried state the agent repeats work and loses the thread across turns.
Human-approval gatepause before any write / send / delete / pay toolKeeps irreversible actions under a person's control; the core safety boundary.
The agent loopwhile not done and steps < MAX: plan -> act -> observeA bounded loop; the MAX cap is what stops runaway behavior when it stalls.
Tool-error handlingreturn {"error": msg} to the model, don't raiseA structured error lets the model correct itself; an exception just ends the task.
Retries & timeoutsper-call timeout; retry idempotent reads with backoffFlaky tools shouldn't hang the loop or duplicate a side effect.
Guardrailsallowlist tools; validate inputs + outputs; sandboxConstrain what the agent can do before trusting what it decides to do.
Observabilitylog every prompt, tool call, args, result, and token countYou can't debug or improve a run you can't replay.
Cost controlscap steps, tokens, and $ / task; alert on breachLoops and growing context turn into a surprise bill fast.
Eval harnessfixed Q&A set scored on every change, as a CI gateTurns 'feels better' into a measured number and catches regressions before deploy.
Prompt-injection defensetreat tool / retrieved output as data, never as instructionsA web page or doc the agent reads can hijack it if the agent obeys text it fetched.

Worked example — a read-only documentation assistant, from task to monitoring

BUILD IT

1
Define the task narrowly and make read-only an explicit rule in the system prompt.
SYSTEM = """You answer questions about OUR product docs.
You may ONLY call search_docs. You cannot write, send, or browse the
open web. If the docs don't cover it, say so. Cite the doc path."""
# scope is now a contract, not a hope
2
Give it exactly one tool, read-only, with a strict schema. No write tools exist, so no approval gate is needed yet.
{"name":"search_docs",
 "description":"Search our docs. Returns up to 5 snippets. Read-only.",
 "input_schema":{"type":"object",
   "properties":{"query":{"type":"string"},
                 "k":{"type":"integer","maximum":5}},
   "required":["query"]}}
3
Run a bounded loop. Validate each call, execute, feed the result back, cap the steps.
for step in range(MAX_STEPS):            # MAX_STEPS = 6
    r = model(messages, tools=[search_docs])
    if r.stop_reason != "tool_use": break
    args = validate(r.tool.input)        # reject bad args
    out  = search_docs(**args)           # read-only call
    messages += [r.tool, {"tool_result": out}]
4
Test on a fixed question set BEFORE deploy. Each case has a question and an expected doc the answer must cite.
cases = load("eval/doc_qa.jsonl")        # 40 hand-labeled Qs
for c in cases:
    a = run_agent(c.q)
    c.pass = c.expected_doc in a.citations and not a.refused
print(f"success {sum(p)}/{len(cases)}  tool_err {errs}/{calls}")
# success 34/40 (85%)  tool_err 1/58  p95 2.3s  $0.004/q
5
Inspect the 6 failures. Replaying logs shows most were the retriever returning the wrong section, not the model.
6
Deploy behind full logging and per-task caps. Monitor the four numbers that matter.
metrics = ["task_success", "tool_error_rate", "p95_latency", "cost_per_task"]
# alert if success < 80% OR cost_per_task > $0.02 OR tool_error_rate > 5%
# week 1: success 88%, tool_err 1.2%, p95 2.1s, $0.005/task
✓
Improve on evidence: reindex the docs that caused the retriever misses, re-run the eval set, keep the change only if success goes up and cost doesn't. Adding a write tool later (e.g. file-a-ticket) means adding the human-approval gate first.

Troubleshooting — symptom → first check → next check

SymptomFirst checkEstablishesNext check
Hallucinated tool args (wrong/invented fields)Log raw call.input and validate it against the schemaBad arguments vs a bad tool resultReturn a schema error for retry; tighten descriptions and add enums/required
Infinite / runaway tool loopCount steps per task in the logsMissing stop condition vs no forward progressEnforce MAX_STEPS and a no-progress detector that breaks the loop
Silent tool failure (agent acts on nothing)Check whether the tool returned an explicit error vs an empty successA swallowed exception vs a genuinely empty resultMake tools return {"error": ...} explicitly; never return empty on failure
Runaway cost / token spendBreak down tokens, steps, and $ per task in the logsLong growing context vs a step explosionCap steps/tokens, prune history, and use a cheaper model for routine steps
Prompt injection (agent obeys fetched content)Did a tool/retrieved result contain instructions the agent then followed?Untrusted content being treated as commandsIsolate tool output as data, re-assert system rules, and gate every write
Low task success on the eval setReplay the failing cases and group them by causeA prompt gap vs a tool/retriever gap vs a model gapFix the dominant failure class, then re-run the full eval before shipping

References

Provider-agnostic; examples use a generic LLM tool-use / function-calling API (Claude Opus 4.x / Sonnet shown generically). Pseudocode is Python-ish; placeholders in CAPS; app.shipreadymetrics.com (scm.cc) is a fictional example host.