LLM-as-a-Judge: Rubrics, Biases and Cohen's Kappa Calibration

At Chappan Express, the support team lead cannot read all 18,000 chatbot replies a day. So she trains two senior agents to review a sample using her checklist. Before trusting them, she grades 50 chats herself and compares.

One senior passes every reply as fine. The other is strict but agrees with her on 45 of 50. Only the second one gets to review chats unsupervised.

Replace the senior agents with a language model and you have LLM-as-a-judge. The checklist is the rubric, and the lead's own grading is the human calibration set.

What is LLM-as-a-judge?

LLM-as-a-judge is an evaluation method in which a language model receives an input, the answer to be graded and a scoring rubric, and returns a score or a preference that stands in for a human rating.

It fills the gap left by the methods in LLM evaluation. Exact match and code rubrics are cheap but cannot tell whether a reply was empathetic or whether it invented a policy. Human review can, but costs too much to run on every change. A judge model sits in between.

For Chappan Help, the judge checks three things code cannot: whether the reply addresses what the customer actually asked, whether its tone suits an upset customer, and whether every claim is supported by the retrieved help article.

Three ways to use a judge model

Pick the style by the question you want answered: "is it good enough?", "which is better?" or "does it match the truth?"

Pairwise comparisons are often more reliable than absolute scores, because judging "which is better" is easier than agreeing on what a 3 out of 5 means. Absolute scores are more convenient for a dashboard. Many teams use pairwise when picking a new prompt and single grading in production.

How to write a good judge prompt

A vague instruction like "rate this answer from 1 to 10" produces vague scores. A good judge prompt is closer to a marking scheme.

1. State the role and the product context. "You are reviewing replies from a food-delivery support bot."
2. Give a small scale with anchors. Pass/fail or 1–3 beats 1–10. Describe what each score means.
3. List concrete checks. "Mentions the 48-hour window", "does not promise a refund amount".
4. Ask for a short reason before the verdict. Reasoning first improves consistency and helps you debug.
5. Force a fixed output format, such as JSON with known keys, as covered in structured output.
6. Run at low temperature so the same input gives the same verdict.

The block below builds such a prompt and safely parses a judge's reply. The judge's text is faked, so it runs anywhere.

import json

RUBRIC = """Score 1-3.
3 = answers the question, states the policy correctly, kind tone.
2 = correct but misses a detail or sounds cold.
1 = wrong policy, invented facts, or ignores the question."""

def judge_prompt(question, context, reply):
    return (f"You review replies from a food-delivery support bot.\n{RUBRIC}\n"
            f"Use only the help article as truth.\n\nHELP ARTICLE:\n{context}\n\n"
            f"CUSTOMER:\n{question}\n\nREPLY:\n{reply}\n\n"
            'Return JSON: {"reason": "<one sentence>", "score": 1|2|3}')

def parse_verdict(raw):
    try:
        v = json.loads(raw[raw.index("{"): raw.rindex("}") + 1])
        return v if v.get("score") in (1, 2, 3) else None
    except ValueError:
        return None

p = judge_prompt("Raita missing, refund?", "Missing items: report within 48 h; wallet refund.",
                 "So sorry! Report it within 48 hours and we'll credit your wallet.")
print(p[:120], "...")
fake_raw = 'Sure. {"reason": "Correct window and wallet, warm tone.", "score": 3}'
print(parse_verdict(fake_raw))
print(parse_verdict("I think it deserves a 3"))   # malformed -> None, retry or flag

Treat unparseable verdicts as errors to retry or count, never as a silent pass. In a real harness you send the prompt through your gateway, as below.

from openai import OpenAI
# continues from the block above: p and parse_verdict are defined there

client = OpenAI(base_url="http://localhost:8000/v1", api_key="YOUR_KEY")  # any OpenAI-compatible endpoint

resp = client.chat.completions.create(
    model="<judge-model>",            # placeholder alias, ideally different from the bot's model
    messages=[{"role": "user", "content": p}],
    temperature=0,
    response_format={"type": "json_object"},
)
verdict = parse_verdict(resp.choices[0].message.content)

Judge biases and how to reduce them

Judge models are not neutral. Research and practice show a few repeatable biases.

• Position bias: in pairwise mode, a judge tends to prefer whichever answer appears first (or second, depending on the model).
• Verbosity bias: longer answers get higher scores even when they add nothing.
• Self-preference: a model tends to favour text written in its own style, so a bot and its judge from the same model family can flatter each other.
• Leniency: many judges rarely give the lowest score unless the rubric spells out failures.

The standard fix for position bias is to ask twice with the order swapped and accept only consistent verdicts. The simulation below shows why.

import numpy as np

rng = np.random.default_rng(5)
n = 400
true_better = rng.choice(["A", "B"], n)        # which reply is really better
gap = rng.uniform(0, 1, n)                      # how obvious the difference is

def biased_judge(first_is_better, gap):
    # picks the truly better reply when the gap is clear, otherwise leans to whichever is shown first
    if gap > 0.4:
        return "first" if first_is_better else "second"
    return "first" if rng.random() < 0.75 else "second"

single, consistent, kept = 0, 0, 0
for t, g in zip(true_better, gap):
    v1 = biased_judge(t == "A", g)               # order A, B
    v2 = biased_judge(t == "B", g)               # order B, A
    pick1 = "A" if v1 == "first" else "B"
    pick2 = "B" if v2 == "first" else "A"
    single += pick1 == t
    if pick1 == pick2:
        kept += 1
        consistent += pick1 == t
print(f"one ordering only : {single / n:.0%} correct")
print(f"swap and agree    : {consistent / kept:.0%} correct on {kept / n:.0%} of pairs kept")

With one ordering, the judge is right about 80% of the time. Requiring both orderings to agree raises accuracy on the kept pairs to about 90%, and the disagreeing pairs become ties or go to a human. For verbosity bias, add "length is not a quality" to the rubric and include short correct answers in your calibration set.

How do you know the judge is right? Cohen's kappa

Before a judge gates releases, compare it with humans on the same cases. Raw percentage agreement is misleading when most answers are good: a judge that always says "pass" would agree 90% of the time with a human who passes 90% of replies.

Cohen's kappa corrects for agreement expected by chance. It is (observed agreement − chance agreement) / (1 − chance agreement). A kappa of 1 is perfect, 0 is no better than chance.

Calibrate once, then re-check whenever you change the rubric or the judge model.

import numpy as np
from sklearn.metrics import cohen_kappa_score

human = np.array([1,1,1,1,1,1,1,1,0,1, 1,1,1,0,1,1,1,1,1,0])   # 1 = pass
lazy  = np.ones(20, dtype=int)                                   # always passes
good  = np.array([1,1,1,1,1,1,1,0,0,1, 1,1,1,0,1,1,1,1,1,1])

def kappa(a, b):
    po = (a == b).mean()
    pe = a.mean() * b.mean() + (1 - a.mean()) * (1 - b.mean())
    return (po - pe) / (1 - pe)

for name, judge in [("always-pass judge", lazy), ("rubric judge", good)]:
    print(f"{name:<18} agreement {(judge == human).mean():.0%}  "
          f"kappa {kappa(human, judge):.2f}  sklearn {cohen_kappa_score(human, judge):.2f}")

h5 = [5, 4, 4, 3, 2, 5, 1, 4, 3, 5]
j5 = [5, 4, 3, 3, 2, 4, 2, 4, 3, 5]
print("1-5 scale, quadratic-weighted kappa:", round(cohen_kappa_score(h5, j5, weights="quadratic"), 2))

The always-pass judge agrees 85% of the time yet has a kappa of zero: it learned nothing. The rubric judge reaches 0.61, just inside the usual "substantial" band. For 1–5 scales, use quadratic-weighted kappa, which punishes a 5-vs-1 disagreement far more than 5-vs-4.

Common rules of thumb treat kappa above about 0.6 as substantial agreement and above 0.8 as strong. Pick your bar before you look at the result, and include at least 50 to 100 human-labelled cases, with plenty of failures.

Common mistakes with LLM judges

• Never calibrating. An uncalibrated judge is an opinion, not a metric.
• Using a 1–10 scale. Judges and humans both cluster scores; small scales with anchors agree better.
• Grading with the same model that wrote the answer. Use a different model family where you can.
• Hiding the context. For faithfulness, the judge needs the retrieved article, not just the answer. See RAG evaluation.
• Ignoring cost. A judge call per test case per change adds up; cache verdicts for unchanged answers.
• Asking one judge question about five qualities. Split into separate checks so failures are traceable.

Key takeaways

• An LLM judge scores or compares replies using a written rubric, scaling open-ended review.
• Use single grading for gates, pairwise for choosing versions, and references when the truth is known.
• Small scales, reasons before verdicts, JSON output and low temperature make judges consistent.
• Swap order for pairwise judging and guard against verbosity and self-preference.
• Measure agreement with humans using Cohen's kappa before trusting the judge. Next, try to break the bot on purpose in red teaming LLM apps.