AI CLI
RAG & Embeddings — retrieval-augmented generation that works
Ground LLM answers in your own data: chunk, embed, store in a vector DB, retrieve top-k by similarity, and generate cited answers that resist hallucination.
Docs-checkedragembeddingsvector-searchretrievalllm
Quick reference — reach for these first
Chunk
split(DOC, size=512, overlap=64)Break docs into passages small enough to embed and retrieve cleanly; overlap preserves cross-boundary context.Embed
embeddings.create(MODEL, input=CHUNK)Map each chunk to a dense vector. Use the SAME model for documents and queries, always.Store
pgvector / FAISS / managed indexPersist vectors + metadata in a vector store with an ANN index for fast cosine similarity search.Retrieve
topk = index.search(embed(Q), k=5)Rank candidates by cosine similarity; add hybrid keyword search and a reranker to lift precision.Generate
LLM(SYSTEM + context + Q) -> [n] answerAnswer ONLY from retrieved passages, each claim cited by passage number, within the context budget.RAG pipeline: index offline, retrieve and cite at query time
Text equivalent: RAG pipeline. Offline indexing row: Docs to Chunk to Embed to Vector DB. Query-time row: Query to Embed to Vector Search, which pulls the top-k most similar chunks from the Vector DB, to Top-k to Prompt plus Context to LLM to a Cited Answer with bracketed citations.
Common tasks
Task · Command / query · Check
| Task | Command / query | Check |
|---|---|---|
| Enable the pgvector extension [prod: DDL] | CREATE EXTENSION IF NOT EXISTS vector; | \dx lists 'vector' among installed extensions |
| Create a table with a vector column | CREATE TABLE docs (id bigserial PRIMARY KEY, content text, content_hash text, embedding vector(1536)); | \d docs shows an 'embedding' column of type vector(1536) |
| Build an ANN index for cosine distance [prod: DDL] | CREATE INDEX ON docs USING hnsw (embedding vector_cosine_ops); | EXPLAIN on a search query shows an Index Scan, not a Seq Scan |
| Chunk a document (size + overlap) | chunks = split_text(DOC, chunk_size=512, chunk_overlap=64) | len(chunks) > 1 and adjacent chunks overlap by ~64 tokens |
| Embed chunks in one batch call | vecs = client.embeddings.create(model='text-embedding-3-small', input=CHUNKS).data | len(vecs) == len(CHUNKS) and len(vecs[0].embedding) == 1536 |
| Insert chunk + embedding rows | INSERT INTO docs (content, content_hash, embedding) VALUES (:CHUNK, :HASH, :EMBEDDING); | SELECT count(*) FROM docs equals the number of chunks |
| Retrieve top-k by cosine similarity | SELECT id, content FROM docs ORDER BY embedding <=> :QUERY_VEC LIMIT 5; | Returns 5 rows; the known-relevant chunk is among them |
| Hybrid search (keyword + vector) | SELECT id FROM docs WHERE to_tsvector(content) @@ plainto_tsquery(:Q) ORDER BY embedding <=> :QUERY_VEC LIMIT 20; | Recall on exact-term/rare-keyword queries beats vector-only |
| Rerank the candidate set | ranked = rerank(model=RERANKER, query=Q, docs=CANDIDATES, top_n=5) | The gold chunk moves into the top 3 after reranking |
| Assemble a grounded prompt | prompt = '\n'.join(f'[{i}] {c}' for i,c in enumerate(TOPK,1)) + '\n\nQUESTION: ' + Q | Prompt contains numbered passages and fits the context budget |
| Generate a cited answer (Claude Opus 4.x / Sonnet) | msg = client.messages.create(model=MODEL, system=SYSTEM, messages=[{'role':'user','content':prompt}]) | Answer cites passages like [1] and adds no uncited claims |
| Detect stale rows and re-embed on change [prod: rewrites rows] | UPDATE docs SET embedding = :NEW_VEC, content_hash = :SRC_HASH WHERE content_hash <> :SRC_HASH; | Edited sources get fresh vectors; hashes match after the run |
| Measure retrieval hit-rate @k | hit = sum(gold in topk(embed(q), k=5) for q,gold in LABELED) / len(LABELED) | hit@5 printed as a fraction on the labeled set (aim > 0.8) |
| Score answer faithfulness with an LLM judge | score = judge(MODEL, question=Q, answer=A, context=TOPK) # 1 if every claim is supported | Groundedness near 1.0; flagged rows contain uncited claims |
Minimal grounded RAG query with citations
AI
1
Problem: answer a question using ONLY retrieved chunks, and cite them — no outside knowledge.
2
Copy-ready system prompt (the grounding contract)
You are a retrieval QA assistant. Answer the QUESTION using ONLY the numbered CONTEXT passages. Cite every claim with its passage number like [1]. If the answer is not in the context, reply exactly: 'Not in the provided context.' Never use outside knowledge.
3
Sample input: the question plus the top-2 retrieved chunks
QUESTION: What is the default dimensionality of OpenAI's text-embedding-3-small? CONTEXT: [1] text-embedding-3-small returns 1536-dimensional vectors by default; the dimensions parameter can shorten them. [2] Cosine similarity is the recommended distance metric for OpenAI embeddings.
4
Generate with a current Claude model (Opus 4.x / Sonnet)
client.messages.create(model=MODEL, system=SYSTEM, messages=[{'role':'user','content':Q_PLUS_CONTEXT}])5
Expected grounded answer — one cited claim, nothing uncited
6
Negative control: ask something absent from the context
7
Evaluate — faithfulness + retrieval hit-rate on a small labeled set
faithful = every_claim_cited_and_supported(answer, context) # LLM judge -> 1/0 hit_at_5 = mean(gold_id in retrieve(q, k=5) for q, gold_id in LABELED)
8
Verify: faithfulness is 1 on the sample and hit@5 clears your bar (e.g. > 0.8). If hit@5 is low, retrieval is the bottleneck — fix chunking/embedding/k before touching the prompt.
Troubleshooting — symptom → first check → next check
| Symptom | First check | Establishes | Next check |
|---|---|---|---|
| Retrieval misses the obviously relevant chunk | Re-run the query with a larger k and print the raw similarity scores | Whether it is a ranking/k problem or the chunk was never indexed well | Lower chunk size / add overlap, confirm query and docs use the SAME embedding model, raise k, then add a reranker |
| Answer ignores the retrieved context and uses model priors | Log the final assembled prompt and confirm the passages are actually in it | Whether context was truncated out or the grounding instruction is too weak | Add a strict 'answer ONLY from context' instruction, put passages near the question, and trim to fit the window |
| Answer cites a source that does not support the claim (hallucinated citation) | Map each [n] back to its passage and verify the claim is present there | Whether the model fabricated support or retrieval returned the wrong chunk | Enforce cite-from-context in the prompt, run a faithfulness judge in CI, and fail the answer on any uncited claim |
| Good answers yesterday, wrong or outdated answers today | Compare the source content_hash to the stored row's hash / updated_at | Whether the index is stale relative to the source of truth | Re-embed changed records on write (or on a schedule) and reindex; never edit a source without re-embedding it |
| Prompt exceeds the context window or token cost spikes | Count tokens of system + passages + question before the call | Whether too many or too-large chunks are being stuffed | Rerank and keep only the top 3-5 passages, shrink chunk size, or summarize passages before stuffing |
| Similarity scores are all near-identical or meaningless | Confirm query and documents were embedded by the same model and the index metric matches (cosine vs L2) | Whether a model/metric mismatch is corrupting the ranking | Use one embedding model everywhere, normalize vectors, and match the index op-class (vector_cosine_ops) to your distance |
References
- OpenAI — Embeddings guide (models, dimensions, cosine)platform.openai.com
- pgvector — Postgres vector store (HNSW, vector_cosine_ops)github.com
- FAISS — similarity search library wikigithub.com
- Anthropic — Citations (grounded, verifiable answers)docs.anthropic.com
- Anthropic — Contextual Retrieval (better RAG recall)anthropic.com
Examples use pgvector + OpenAI text-embedding-3-small (1536 dims, cosine) and a current Claude model (Opus 4.x / Sonnet); swap in FAISS or a managed index and your own embedding model as needed. The query and documents must always be embedded by the same model.