When should you use an LLM judge instead of a code check?

When code cannot see the failure, such as tone or grounding

A judge agrees with people 90% of the time, but 90% of replies pass. What should you check?

Kappa, because a judge that always says pass would also get 90%

Which judge setup is steadiest?

Same family as the app, temperature 0.7

Any setup, if the rubric is long

LLM-as-Judge and Calibration

By the Syncfusion DevTutorials team · Last updated October 27, 2026

Some failures need a model to grade them. Learn to write a judge rubric, label replies yourself, measure the judge's agreement with people, and catch its biases before you trust its scores.

LLM-as-Judge and Calibration

Some things are hard to check with code. Is this reply helpful? Is it grounded in the help page? Does it sound rude?

So the team adds an LLM judge: a second model that reads each reply and says pass or fail.

They give it a short prompt and run it on v1 and v2. Watch the result.

The judge passes v1 on 174 of 200 chats and v2 on 179 of 200. It agrees with the old vibe: v2 is better.

But people's labels say v1 is right on 184 chats and v2 on only 164. The judge liked v2's long, warm replies and missed its broken refund rule.

Simple lesson: the judge is a model too. Check it before you trust it.

A Real-World Example

Think about a new referee.

Before a referee runs a big match, they watch recorded games next to an experienced referee. They call each play. Then they compare calls.

If they agree on almost every call, the new referee is ready. If not, they train more.

Calibrating a judge is the same. You label replies yourself, the judge labels the same replies, and you compare.

Simple lesson: a judge earns trust by agreeing with people, not by sounding sure.

What You Will Learn

In this lesson, you will learn:

What an LLM judge is, and when you need one.
How to write a rubric that names exact checks.
How to calibrate a judge against your own labels, with kappa.
How to find verbosity, position and self-preference bias.
Which settings make a judge steady: temperature, model family and averaging.

TL;DR

LLM judge = a separate model that scores an answer against a written rubric.

Rubric = the instructions the judge gets: what makes a reply pass or fail.

Calibration = checking the judge against labels from people, before you use its scores.

Label at least 40 replies yourself. Run the judge on the same ones. Measure agreement with kappa. Aim for 0.6 or more.

Then probe for bias: pad the reply, swap the order, and try another model family. A good judge does not change its verdict.

A judge is a model too. Check it against your own labels before you trust its scores.

Cheat Sheet

Setting	Use	Why
Rubric	Name exact checks, say "ignore length and tone"	Vague rubrics add noise and bias
Temperature	0	Same verdict every run
Model family	Different from the app's model	Removes self-preference
Runs per reply	1, or average 3 to 5 for high-stakes scores	Averaging shrinks noise
Calibration	40+ human labels, kappa 0.6 or more	Proves the judge agrees with people

When You Need a Judge

Last lesson's rule still holds: use the cheapest grader that catches the failure.

A judge is for failures code cannot see:

Is the answer grounded in the help page, in different words?
Is the reply helpful, or does it dodge the question?
Is the tone rude or cold?
Does a summary mean the same as the gold summary?

If code can catch it, use code. A judge is slower, costs a model call per reply, and can be wrong.

The judge in this course is simulated

The Judge lab uses a judge simulator. It behaves like a real judge, with its biases written down: longer replies get a bonus, the first answer gets a bonus, its own model family gets a bonus, and vague rubrics and high temperature add noise.

It only notices a failure the rubric tells it to look for. You can turn on a real small model as the judge in Settings to compare. A real judge never grades your exercises.

Calibrate Against People

First, people label the replies. Then the judge labels the same replies. Then you compare.

Kappa (Cohen's kappa) = a number that measures how much two labellers agree, beyond what luck alone would give. 1 is full agreement. 0 is no better than chance.

Plain agreement can fool you. If 90% of replies pass, a judge that always says "pass" agrees 90% of the time. Kappa corrects for that.

With the vague rubric, the judge reaches kappa 0.41. That is below the bar of 0.6.

With a specific rubric, temperature 0 and another model family, it reaches kappa 0.76 here. That passes the bar. It is not perfect, because a few replies are borderline and even a careful judge calls them wrong. Real judges after calibration usually land between 0.6 and 0.8 too.

Here is kappa in code, for ten replies.

Agreement and kappa for ten labels

// true = pass, false = fail
const people: boolean[] = [true, true, false, false, true, false, true, true, false, true];
const judge: boolean[] = [true, true, true, false, true, false, true, true, true, true];

const n = people.length;
let agree = 0;
for (let i = 0; i < n; i++) if (people[i] === judge[i]) agree++;

const peoplePass = people.filter(Boolean).length / n;
const judgePassRate = judge.filter(Boolean).length / n;
const observed = agree / n;
const byChance = peoplePass * judgePassRate + (1 - peoplePass) * (1 - judgePassRate);
const kappa = (observed - byChance) / (1 - byChance);

console.log(`agreement ${Math.round(observed * 100)}%`);
console.log(`kappa ${Math.round(kappa * 100) / 100}`);

Expected output:

agreement 80%
kappa 0.55

80% agreement sounds good. Kappa says the judge is only a bit better than chance, because it says "pass" almost every time.

Try this: change the judge's two wrong "true" values to false. Predict the kappa, then run it.

Simple lesson: measure agreement with kappa, not with a plain percentage.

Probe for Bias

A judge can agree with people on average and still have habits. Probes find them.

Probe = a small change that should not change the verdict. If it does, the judge has a bias.

Verbosity bias = the judge prefers longer answers. Padding replies with a polite paragraph flipped 18 of 200 verdicts with the vague rubric.

Position bias = in a side-by-side, the judge prefers whichever answer comes first.

Self-preference bias = the judge prefers replies written by its own model family.

Fixes that work

Write "ignore length and tone" in the rubric. Name the exact checks.
In side-by-side judging, judge twice with the order swapped. Keep a verdict only if both agree.
Use a judge from a different model family than the app.

Simple lesson: if a change that should not matter changes the verdict, the judge has a bias.

Noise and Temperature

A judge at a high temperature can give the same reply different scores on different runs.

Your eval score should not wobble because the judge rolled differently. Set the judge's temperature to 0. For scores that decide a release, average three to five runs.

Where It Is Used

Grading answers for helpfulness, grounding or tone, where code cannot see the failure.
Online evals on a sample of live traffic (a later lesson).
Comparing two versions side by side, with the order swapped.

Do's and Don'ts

Do label 40 or more replies yourself before trusting a judge.
Do measure kappa and aim for 0.6 or more.
Do name exact checks in the rubric and say "ignore length".
Do use temperature 0 and a different model family.
Don't use a judge where code can see the failure.
Don't trust a plain agreement percentage.
Don't change the rubric and the app in the same step.

Common mistakes

Mistake	Wrong idea	Correct idea	What goes wrong
Vague rubric	"Is this a good answer?" is enough.	Name the facts, the rules and the citation to check.	The judge grades tone and misses broken rules.
No calibration	"The judge sounds sure, so it is right."	Compare with 40+ human labels and measure kappa.	You ship decisions based on a wrong grader.
Trusting plain agreement	"80% agreement is great."	Use kappa, which corrects for luck.	A judge that always says pass looks good.
Same model family	"Use our app's model as the judge, it is cheap."	Judge with a different model family.	The judge favours its own style.
High temperature	"Temperature does not matter for grading."	Temperature 0, average runs for big decisions.	The same reply passes one run and fails the next.

Check Your Understanding

Exercises

What to Remember

An LLM judge grades what code cannot see. It is a model, so it can be wrong.
Write a rubric that names exact checks and says "ignore length and tone".
Calibrate: label 40+ replies yourself and measure kappa. Aim for 0.6 or more.
Probe for verbosity, position and self-preference bias.
Use temperature 0, a different model family, and average runs for big decisions.

Next: Statistics for Evals, how to tell whether a 2-point gain is real or just noise.