LLM Evaluation Methods: Metrics, Golden Sets and a Python Harness

Think of the head chef at Chappan Express tasting dishes before a new menu launches. She does not taste one plate and declare the menu ready. She tastes every dish, checks each against a written standard (salt, temperature, portion), and compares it with the old recipe.

If the paneer is better but the dal is now too salty, the menu does not launch yet.

Evaluating an LLM app works the same way. The dishes are test questions, the written standard is your scoring method, and the old recipe is the version currently live.

What is LLM evaluation?

LLM evaluation is the practice of running an LLM application on a fixed set of inputs and scoring its outputs against agreed criteria, so that quality can be measured, compared across versions and tracked over time.

There are two kinds. Model evaluation compares raw models on public benchmarks. Application evaluation tests your own system, with your prompt, retriever and tools, on your own users' questions. Benchmarks help you shortlist a model; only application evaluation tells you if Chappan Help is ready.

This lesson is about application evaluation. It continues the LLMOps story from what LLMOps is.

Why is evaluating LLMs hard?

Three things make it harder than scoring a classifier.

• Many correct answers. "48 hours" and "two days" can both be right.
• Several qualities at once. An answer can be correct but rude, or polite but invented.
• Randomness. With sampling turned on, the same question can produce different answers. Fix temperature low for evaluation runs, as explained in LLM generation parameters, or run each case more than once.

Which LLM evaluation methods should you use?

Each method trades cost for the ability to judge open-ended answers. Good evaluation suites mix them.

Move right only when the method on the left cannot judge the question.

BLEU and ROUGE, borrowed from translation and summarisation research, also count overlapping words. They correlate poorly with what support users care about, so most app teams use them rarely.

Reference-based metrics: exact match and token F1

When a question has a short known answer, compare against a reference. Exact match asks whether the normalised strings are identical. Token F1 gives partial credit for shared words, using precision and recall over tokens.

Normalising first matters: lowercase, strip punctuation, and collapse spaces. Otherwise "48 Hours." fails against "48 hours".

import re
from collections import Counter

def norm(s):
    return " ".join(re.sub(r"[^a-z0-9 ]", " ", s.lower()).split())

def exact_match(pred, ref):
    return float(norm(pred) == norm(ref))

def token_f1(pred, ref):
    p, r = norm(pred).split(), norm(ref).split()
    common = sum((Counter(p) & Counter(r)).values())
    if common == 0:
        return 0.0
    precision, recall = common / len(p), common / len(r)
    return 2 * precision * recall / (precision + recall)

cases = [
    ("Refund window for missing items?", "48 hours.", "48 hours"),
    ("Where does a refund go first?", "To your Chappan wallet", "chappan wallet"),
    ("Can I cancel after pickup?", "No", "no"),
    ("Bank refund time?", "about 5 to 7 working days", "5-7 working days"),
]
for q, pred, ref in cases:
    print(f"{q:<34} EM={exact_match(pred, ref):.0f}  F1={token_f1(pred, ref):.2f}")

The wallet answer is clearly correct yet scores zero on exact match, and token F1 only gives it 0.67 because "to your" counts as extra words. The bank answer gets 0.80 for the same reason. Every reference metric has blind spots like this, which is why you read failures, not just averages.

Rubric scoring for open-ended answers

Most Chappan Help replies are full sentences. For those, write a rubric: a short list of checks that a good answer must pass, each with a weight. Many checks can be plain code.

For the refund question, a good answer must mention the 48-hour window, say the money goes to the wallet, never promise a specific rupee amount the policy does not give, and stay short.

import re

RUBRIC = [
    ("mentions 48-hour window", 0.35, lambda a: bool(re.search(r"48 ?hours|two days", a, re.I))),
    ("names the wallet",        0.25, lambda a: "wallet" in a.lower()),
    ("no invented amount",      0.25, lambda a: not re.search(r"(rs\.?|₹) ?\d", a, re.I)),
    ("under 60 words",          0.15, lambda a: len(a.split()) <= 60),
]

def score(answer):
    passed = [(name, w) for name, w, check in RUBRIC if check(answer)]
    return sum(w for _, w in passed), [n for n, _, _ in RUBRIC if n not in dict(passed)]

answers = {
    "v7": "Sorry about that! Report missing items within 48 hours and we refund to your Chappan wallet.",
    "v8": "So sorry! We'll refund Rs 250 to your wallet right away.",
    "v9": "Missing items can be reported within two days. The refund reaches your wallet within 24 hours.",
}
for version, a in answers.items():
    s, failed = score(a)
    print(f"{version}: {s:.2f}  failed: {failed or 'none'}")

Version v8 sounds friendlier but invents an amount and drops the time limit. A rubric catches that instantly and for free. For qualities code cannot check, such as empathy or whether an answer is faithful to retrieved text, you hand the rubric to a model. That is the subject of LLM-as-a-judge.

How to build a golden evaluation set

A golden set is your fixed list of test inputs with references or rubrics. It is the most valuable asset in LLM evaluation, more than any tool.

1. Pull real questions. Sample 200 recent support chats, not invented ones. Remove names, phones and order IDs.
2. Tag each with a category (order status, refund, cancel, allergen, out of scope) so you can see where a change hurts.
3. Write the reference or rubric with someone who knows the policy, not only the developer.
4. Add hard cases on purpose: Hinglish, typos, two questions in one message, angry customers, questions the bot must refuse.
5. Freeze and version it. Change the set only through a reviewed commit, or scores stop being comparable.
6. Grow it from production. Every thumbs-down that reveals a new failure becomes a new case.

One harness, many scorers, results by category, and a comparison with the live champion.

Is the difference real? Comparing two versions

Suppose prompt v8 passes 34 of 40 cases and v7 passes 31. Is v8 better, or was it luck? With small sets, a few cases flipping can move the score several points.

A paired bootstrap answers this. Resample the same cases with replacement many times and look at the spread of the difference.

import numpy as np

rng = np.random.default_rng(3)
# 40 golden cases, 1 = pass. Both versions agree on 33 cases.
v7 = np.array([1] * 29 + [0] * 4 + [1] * 2 + [0] * 5)   # 31 passes
v8 = np.array([1] * 29 + [0] * 4 + [0] * 2 + [1] * 5)   # 34 passes
n = len(v7)

diffs = []
for _ in range(5000):
    idx = rng.integers(0, n, n)                   # same cases for both versions
    diffs.append(v8[idx].mean() - v7[idx].mean())
lo, hi = np.percentile(diffs, [2.5, 97.5])
print(f"v7 pass rate {v7.mean():.0%}, v8 pass rate {v8.mean():.0%}")
print(f"95% interval for v8 - v7: {lo:+.1%} to {hi:+.1%}")
print("Clear win" if lo > 0 else "Not distinguishable from noise yet")

When the interval crosses zero, the honest answer is "no clear difference". Either add more cases or accept the version that is cheaper or safer. Teams that skip this step chase noise for weeks.

Running evaluations with a framework

Hand-written harnesses teach the idea. Frameworks save time once the suite grows. Here is a model-graded metric in DeepEval, written against its current documentation; it needs an API key for the judge model, so it does not run in the browser.

from deepeval import evaluate
from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCase, SingleTurnParams

policy_check = GEval(
    name="Refund policy correctness",
    evaluation_steps=[
        "Check that the answer states the 48-hour reporting window for missing items.",
        "Penalise any refund amount that is not in the expected output.",
        "Penalise answers that promise a bank refund when the policy says wallet first.",
    ],
    evaluation_params=[SingleTurnParams.INPUT, SingleTurnParams.ACTUAL_OUTPUT,
                       SingleTurnParams.EXPECTED_OUTPUT],
    threshold=0.7,
)
case = LLMTestCase(
    input="My raita was missing from yesterday's order. Refund?",
    actual_output="Sorry! Missing items reported within 48 hours are refunded to your wallet.",
    expected_output="Report within 48 hours; refund goes to the Chappan wallet within 24 hours.",
)
evaluate(test_cases=[case], metrics=[policy_check])

For retrieval-specific metrics such as context recall and faithfulness, see evaluating RAG systems. Agents add tool-call checks, covered in AI agent evaluation.

Common mistakes in LLM evaluation

• Testing on invented questions. Real users write differently. Start from logs.
• One average number. A 2-point overall gain can hide a 15-point drop on allergen questions. Always report by category.
• Changing the golden set and the prompt together. Then you cannot tell which change moved the score.
• Sampling at high temperature during evals. Results will not repeat. Lower temperature or average several runs.
• Trusting a judge you never checked. Measure its agreement with humans first.
• Skipping refusals. Cases where the right answer is "I can't help with that" are part of quality.

Key takeaways

• Evaluate your application, not just the model, on a versioned golden set of real questions.
• Use exact match and F1 for short answers, code rubrics for policy and format, and judges or humans for the rest.
• Report per category and check whether differences exceed noise before celebrating.
• Grow the golden set from production failures.
• Next: let a model do the grading in LLM-as-a-judge.