RAG Evaluation: Metrics to Test Retrieval and Answer Quality

At the end of a semester, a professor at an Indore college checks an open-book exam. She asks two separate questions about every answer. Did the student open the right pages of the textbook? And did they write what those pages actually say, without adding made-up facts?

A student can fail either way. One opens the wrong chapter and writes nonsense confidently. Another opens the right page and still misquotes it.

Grading a RAG system works the same way. You check the "looking up" and the "writing" separately, because they fail for different reasons and need different fixes.

What is RAG evaluation?

Mapping the exam: the textbook pages the student opened are the retrieved chunks, the written answer is the LLM's output, the answer key is your labelled golden set, and the professor's two checks are retrieval metrics and generation metrics.

RAG evaluation is the practice of measuring a retrieval-augmented generation system with a fixed set of test questions, scoring both whether it retrieves the right context and whether its answers are faithful to that context, relevant and correct.

Every lesson in this series ended with "measure it": chunk sizes in RAG chunking strategies, fusion settings in hybrid search and re-ranking, and HyDE in HyDE explained. This lesson shows how, on the same fictional KIT college handbook.

The two halves of RAG quality

Each metric checks one relationship. Context metrics test retrieval; faithfulness and relevance test generation; correctness needs a reference answer.

The diagnosis follows from the pattern. Low retrieval scores mean fix chunking, search or re-ranking. Good retrieval with low faithfulness means fix the prompt or the model.

Retrieval metrics: hit rate, recall, precision and MRR

Take one question that needs two chunks: "Can I get a refund if I withdraw in week two, and when is it credited?". The retriever returns five chunks, and the two you need sit at ranks 2 and 4.

def hit_at_k(ranked, relevant, k):
    return int(any(d in relevant for d in ranked[:k]))

def precision_at_k(ranked, relevant, k):
    return sum(d in relevant for d in ranked[:k]) / k

def recall_at_k(ranked, relevant, k):
    return sum(d in relevant for d in ranked[:k]) / len(relevant)

def reciprocal_rank(ranked, relevant):
    for rank, d in enumerate(ranked, start=1):
        if d in relevant:
            return 1 / rank
    return 0.0

# "Can I get a refund if I withdraw in week two, and when is it credited?"
relevant = {"fees#refund", "fees#credit"}           # both chunks are needed
ranked = ["fees#late", "fees#refund", "lib#fine", "fees#credit", "att#min"]

for k in (1, 3, 5):
    print(f"k={k}: hit={hit_at_k(ranked, relevant, k)}  "
          f"precision={precision_at_k(ranked, relevant, k):.2f}  "
          f"recall={recall_at_k(ranked, relevant, k):.2f}")
print("reciprocal rank:", reciprocal_rank(ranked, relevant))

At k = 3 the hit rate is 1, yet recall is only 0.50: the credit-timing chunk is missing, so the answer will be half right.

At k = 5 recall reaches 1.00 but precision is 0.40, meaning three of five chunks are noise in the prompt. The reciprocal rank is 0.5 because the first relevant chunk sits at rank 2.

Mean reciprocal rank (MRR) is the average of that value over all questions. Use hit rate for single-fact questions, recall when answers need several chunks, and MRR when position matters, for example before a small top-k.

Comparing two retrievers on a golden set

A golden set is a list of real questions, each labelled with the chunk IDs that answer it. Once you have one, comparing two designs takes seconds. Here a word-level TF-IDF retriever competes with character n-grams, which tolerate variants like "refund" and "refunds":

import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

chunks = {
    "fees#late":    "Semester fees are due by 15 July and a late fee of 100 rupees per day applies.",
    "fees#refund":  "Withdrawal in the first 15 days of classes earns a 50 percent refund.",
    "fees#credit":  "Approved refunds are credited to your bank account within 10 working days.",
    "att#min":      "Each subject needs 75 percent attendance to sit the end-semester exam.",
    "att#medical":  "A medical certificate must reach the office within 7 days of rejoining.",
    "exam#reval":   "Revaluation of a paper costs 300 rupees and must be requested within 7 days of the result.",
}
golden = [  # (question, ids of chunks that answer it)
    ("How many days until my refund is credited?", {"fees#credit"}),
    ("Do I get any money back if I withdraw in week two?", {"fees#refund"}),
    ("What is the penalty for paying semester fees late?", {"fees#late"}),
    ("What attendance do I need to write exams?", {"att#min"}),
    ("When must I submit a doctor's note after illness?", {"att#medical"}),
    ("How do I get my answer sheet rechecked?", {"exam#reval"}),
]
ids, texts = list(chunks), list(chunks.values())

def evaluate(vectorizer, k=3):
    vec = vectorizer.fit(texts)
    hits, rr = [], []
    for q, relevant in golden:
        scores = cosine_similarity(vec.transform([q]), vec.transform(texts))[0]
        order = [ids[i] for i in np.argsort(-scores)]
        rank = next((r for r, d in enumerate(order, 1) if d in relevant), None)
        hits.append(rank is not None and rank <= k)
        rr.append(1 / rank if rank else 0)
    return np.mean(hits), np.mean(rr)

for name, v in {"word TF-IDF": TfidfVectorizer(stop_words="english"),
                "char 3-5 grams": TfidfVectorizer(analyzer="char_wb", ngram_range=(3, 5))}.items():
    hit, mrr = evaluate(v)
    print(f"{name:<15} hit@3 = {hit:.2f}   MRR = {mrr:.2f}")

Character n-grams win clearly here: hit@3 rises from 0.67 to 1.00 and MRR from 0.56 to 1.00. Be suspicious of a perfect score on six questions, though.

"Rechecked" probably matched "revaluation" through shared fragments like "re", which is luck, not understanding. With 6 questions, one lucky match moves hit rate by 17 points. That is why you need 30 to 50.

Generation metrics: faithfulness and answer relevance

Good retrieval does not guarantee a good answer. The model can still add facts that are not in the context. Faithfulness, also called groundedness, is the share of claims in the answer that the retrieved context supports.

The usual method has three steps: split the answer into individual claims, check each claim against the context, and divide supported claims by total claims. This toy checker uses word overlap and numbers instead of an LLM, to show the shape of the calculation:

import re

context = ("Withdrawal in the first 15 days of classes earns a 50 percent refund. "
           "Approved refunds are credited to your bank account within 10 working days.")
answer = ("You get a 50 percent refund if you withdraw within 15 days. "
          "The money is credited within 10 working days. "
          "You can also claim a full refund of the hostel deposit within 30 days.")

def numbers(text):
    return set(re.findall(r"\d+", text))

def content_words(text):
    stop = {"you", "the", "a", "if", "is", "of", "in", "to", "can", "get", "also", "your", "within"}
    return {w for w in re.findall(r"[a-z]+", text.lower()) if w not in stop}

ctx_nums, ctx_words = numbers(context), content_words(context)
claims = re.split(r"(?<=\.)\s+", answer)
supported = 0
for claim in claims:
    nums_ok = numbers(claim) <= ctx_nums                     # every number appears in context
    overlap = len(content_words(claim) & ctx_words) / len(content_words(claim))
    ok = nums_ok and overlap >= 0.5
    supported += ok
    print(f"{'SUPPORTED  ' if ok else 'UNSUPPORTED'} overlap={overlap:.2f} | {claim}")
print(f"Faithfulness = {supported}/{len(claims)} = {supported / len(claims):.2f}")

The first two claims are backed by the context. The third invents a hostel deposit rule and a 30-day window that appear nowhere in the retrieved text, so faithfulness is 0.67. The claim might even be true somewhere in the handbook, but the model did not get it from the context, which makes it a hallucination risk.

Word overlap is crude: it cannot tell "50 percent refund" from "no 50 percent refund". Real systems use an LLM judge for this check.

Using an LLM as the judge

An LLM judge receives the context and one claim and returns a verdict. Keep the prompt narrow, ask for a fixed format, and test it against human labels before trusting it. LLM-as-a-judge covers the biases to watch for.

You are checking whether a claim is supported by the context.
Answer with exactly one word: SUPPORTED, CONTRADICTED or NOT_FOUND.

Context:
{retrieved_chunks}

Claim:
{one_sentence_from_the_answer}

Answer relevance is judged in a similar way: does the answer address the question, or does it wander? A faithful answer about hostel curfew is still useless if the student asked about refunds.

Open-source libraries package these metrics. Ragas, for example, offers faithfulness, answer relevancy, context precision and context recall. Their APIs change between versions, so check the current docs before copying examples.

How to build a RAG evaluation set

Evaluation is a loop: one fixed question set, one change at a time, and failure review after each run.

1. Collect real questions. Use help-desk emails, chat logs and questions from students, not only ones you invent. Include Hinglish and typos if users write that way.
2. Cover the hard cases. Add multi-part questions, questions using codes like "ATKT", and questions the handbook cannot answer.
3. Label each question with the chunk IDs that answer it and a short reference answer. For unanswerable questions, the reference is "I don't know".
4. Run the pipeline and store everything: retrieved IDs, scores, prompt and answer.
5. Score retrieval and generation separately using the metrics above.
6. Read the failures. Twenty minutes reading wrong answers teaches more than any average.
7. Change one thing, rerun, compare. Keep the change only if the numbers improve and nothing important regresses.

LLMs can draft extra questions from your chunks to grow the set, but review them; generated questions tend to copy the chunk's wording and make retrieval look better than it is.

Common mistakes in RAG evaluation

• Only testing end-to-end answers. When the score drops you cannot tell whether retrieval or generation broke.
• Testing on questions you wrote while reading the documents. They reuse the documents' words and hide vocabulary mismatches.
• Too few questions. Six questions make every change look dramatic; 30 to 50 is a sensible minimum.
• Trusting an unvalidated judge. Compare the judge's verdicts with human labels on 20 to 30 examples first.
• Never re-running. Re-run after every change to chunking, embeddings, prompts or models, and before every release.

For evaluating LLM outputs more broadly, beyond RAG, see LLM evaluation methods.

Key takeaways

• Evaluate retrieval and generation separately so you know what to fix.
• Hit rate, recall, precision and MRR measure retrieval against labelled chunk IDs.
• Faithfulness and answer relevance measure generation, usually with a validated LLM judge.
• Build a golden set of 30 to 50 real questions, including unanswerable ones.
• Change one thing at a time and rerun the same set every time.

Evaluation is what separates a demo from a project an interviewer takes seriously. A Generative AI course in Indore that teaches RAG evaluation hands-on is offered at Debugshala, and you can benchmark yourself with timed MCQs on Debugshala Test.