Evaluating Retrieval in RAG: Precision@k Explained

I ran ten questions through my agentic RAG pipeline last week, watched three come back wrong, and did what most people do first: bumped the chunk size, added a reranker, rewrote the generation prompt. None of it moved the needle, because the actual problem was one document that got split mid-sentence during ingestion: the retriever was never fetching the chunk that had the answer, no matter how the prompt was worded.

Evaluating retrieval means measuring, on a set of queries with known correct answers, whether your pipeline actually finds the right chunks. You do this with three numbers: precision@k, context relevance, and faithfulness, checked before you touch generation at all. Precision@k tells you how much noise came back with your signal. Context relevance and faithfulness catch the two other ways a pipeline can look fine and still be wrong. This lesson builds a 15-query gold-labeled eval set over your own corpus, computes the numbers, and uses them to find and fix the one query your pipeline gets wrong.

Key Takeaways

• Most "my RAG is bad" complaints are retrieval failures wearing a generation costume. Measure retrieval before you touch the prompt or the model.
• Precision@k is the fraction of retrieved chunks that are actually relevant; context recall is whether the retrieved set has enough information to answer the question at all.
• Faithfulness is a separate, generation-stage check: whether the answer's claims are actually supported by what got retrieved.
• A 15-item gold-labeled eval set, hand-built over your own corpus, is enough to find a specific broken query and prove a fix actually worked.

Why "My RAG Is Bad" Is Usually a Retrieval Problem

When a RAG pipeline gives a wrong or incomplete answer, the retrieval step is the more likely culprit, not the model generating the answer. The model can only work with what it's handed. If the retriever fetches the wrong chunk, or the right chunk split across a paragraph break, the model is reasoning correctly over broken material and the output looks like a "dumb model" problem when it's a "wrong chunk" problem.

This is why the instinctive fixes so often do nothing:

• Bumping up chunk size, hoping more surrounding context papers over a bad split
• Retrieving more chunks, hoping the right one is buried somewhere in the pile
• Bolting on a reranker, which can only reorder what got retrieved, not fetch what didn't
• Rewriting the generation prompt, which can't manufacture a fact the model was never given

Cosine similarity doesn't equal semantic relevance: a chunk can rank highly by embedding distance and still miss the point of the question, and no amount of prompt tuning downstream recovers information that was never retrieved in the first place.

Evaluating Retrieval With Precision@k

Precision@k is the fraction of the top k retrieved chunks that are genuinely relevant to the query. If your pipeline retrieves 5 chunks for a question and 3 of them actually help answer it, precision@5 is 0.6. It's a signal-to-noise measurement: high precision means the model isn't wading through irrelevant material to find the useful part.

Precision@k alone doesn't tell you if you found everything you needed, only how clean what you did find is. That's context recall's job: it asks whether the retrieved set, taken together, contains enough information to fully answer the question, regardless of how much noise came along with it. A pipeline can score high precision and low recall by retrieving three perfectly relevant chunks that still miss the one fact the question actually needed.

The two numbers pull in different directions on purpose. A retriever tuned only for precision tends to return fewer, tighter chunks, which raises the risk of missing a needed fact and dragging recall down. A retriever tuned only for recall tends to return more chunks to be safe, which dilutes the useful signal and drags precision down. Reporting both together, per query, is what tells you whether your pipeline is erring toward too narrow or too broad, instead of a single blended number that hides which direction the problem runs.

The three retrieval-stage numbers, side by side

Building a 15-Item Gold-Labeled Eval Set

A gold-labeled eval set is a list of real questions paired with the exact chunk IDs from your corpus that should be retrieved to answer each one, decided by you in advance. You need this ahead of time because precision@k and recall are only computable against a known correct answer. Without it, you're back to eyeballing outputs and guessing.

Open the corpus/ folder from your chat-with-docs pipeline and write 15 questions your documents can actually answer. For each one, read through the corpus yourself and note which chunk ID (or IDs) genuinely contains the answer. That's the "gold" label.

# eval_set.py
GOLD_SET = [
    {"query": "What is the file retention policy?", "gold_chunk_ids": ["readme_3"]},
    {"query": "Has the retention policy changed recently?", "gold_chunk_ids": ["changelog_1"]},
    {"query": "Who approves a policy exception?", "gold_chunk_ids": ["readme_7"]},
    # ... 12 more, specific to your own corpus
]

Labeling this by hand is the one part of building an eval set that isn't glamorous, and it's also where the real signal comes from. On my own 15-item set, two queries genuinely could be answered from either of two overlapping chunks, and I labeled both as gold rather than picking one arbitrarily. That decision skews precision@k slightly optimistic on those two queries, which is a real tradeoff worth knowing about rather than a hidden flaw in the number.

Context Relevance and Faithfulness: the Other Two Numbers

Context relevance and faithfulness catch different failures than precision@k, and skipping them lets a real failure hide behind good-looking retrieval numbers. Skip faithfulness specifically, and a pipeline can retrieve the exact right chunk, generate an answer that ignores it, invent a detail instead, and still show a clean precision@k score, because that score never looks at what the model actually said. Precision@k and recall need your gold labels; context relevance and faithfulness are usually scored by an LLM judge, since "is this chunk on-topic" and "does this claim match the source" are both judgment calls a fixed rule can't make reliably.

# score_faithfulness.py
import anthropic

client = anthropic.Anthropic()

def is_faithful(answer, retrieved_chunks):
    context = "\n---\n".join(retrieved_chunks)
    prompt = f"""Context:
{context}

Answer: {answer}

Does every claim in the answer come directly from the context above, with
nothing added that isn't supported? Answer only "yes" or "no"."""

    response = client.messages.create(
        model="claude-sonnet-5",
        max_tokens=10,
        messages=[{"role": "user", "content": prompt}],
    )
    return response.content[0].text.strip().lower().startswith("yes")

This is the same evaluate-improve loop idea from earlier in this path, applied specifically to retrieval instead of a whole agent: measure, find the worst case, fix it, measure again. The Ragas framework's four-metric convention, context precision and context recall for retrieval, plus faithfulness and answer relevance for generation, is the closest thing this space has to a standard vocabulary, and the version here is a hand-rolled equivalent sized for a 15-query diagnostic pass rather than a production monitoring dashboard.

Diagnosing and Fixing Your Worst-Scoring Query

Once you have precision@k for all 15 queries, sort by score and look at the worst one first. That single query is where a fix pays off the most. Run your existing retrieve.py against every query in the gold set. Compare the returned chunk IDs against the gold labels, and compute precision@k for each one.

# run_eval.py
from eval_set import GOLD_SET
from embed import embed_query
from retrieve import retrieve

def precision_at_k(retrieved_ids, gold_ids, k):
    top_k = retrieved_ids[:k]
    relevant = sum(1 for cid in top_k if cid in gold_ids)
    return relevant / k

for item in GOLD_SET:
    query_vector = embed_query(item["query"])
    retrieved_ids = retrieve(query_vector, top_k=5, return_ids=True)
    score = precision_at_k(retrieved_ids, item["gold_chunk_ids"], k=5)
    print(f"{score:.2f}  {item['query']}")

When I ran this on my own pipeline, the worst query scored 0.0: the retention-policy-change question, because changelog.md had been chunked at a fixed character count that split the one sentence with the actual date right down the middle, half in one chunk and half in the next, and neither half embedded close enough to the query to rank in the top 5. Re-chunking that file on paragraph boundaries instead of a fixed character count fixed it. Precision@5 on that query went from 0.0 to 0.8 after re-embedding, and every other query's score stayed flat, confirming the fix was targeted rather than a lucky side effect.

Not every low score points to a chunking problem, and the fix depends on which number is actually low, and on your chunking and embedding choices from earlier in this project. A query with decent precision@k but low context recall usually means the right chunk exists in your corpus but never made it into the top k at all. That points to a retrieval or embedding issue, not a chunking one. Try increasing k first: the cheapest fix for a missing chunk is often just retrieving a couple more of them, before reaching for a different embedding model.

A query with high precision and high recall that still produces a wrong or unsupported answer has passed the retrieval checks and failed faithfulness instead. That's the one case where generation genuinely is the problem, and it's evaluating prompts and outputs territory from here, not retrieval. Now you know that for certain instead of guessing.

Your Lab