Evaluator Agents

By the Syncfusion DevTutorials team · Last updated October 22, 2026

Why an agent that grades its own work passes everything, and how a separate evaluator with a concrete rubric catches skipped tests, rewritten descriptions, unrelated edits and features that never turn on. Covers the planner, generator and evaluator roles for frontend, backend and UI design reviews.

Evaluator Agents

Your long project now has a plan, sessions and an end-to-end check.

One question is left. Who decides that a feature is done?

So far, the agent that built the feature also judged it. That is a weak spot.

Evaluator = a separate agent that judges finished work. It did not build that work.

This lesson splits the work into three roles. It also shows how to write the evaluator's checklist and test it.

For example, six pieces of work wait for review:

• One is really done.
• One has an Export button that is never turned on.
• One has a failing test that was switched off.
• One has a feature description that was made easier.
• One changed 14 files nobody asked about.
• One has failing unit tests.

A Real-World Example

Think about a school exam.

If students marked their own exams, almost everyone would pass.

They are not cheating. They know what they meant to write.

So a teacher marks the exam. The teacher uses a marking scheme: "1 point if the answer has units."

The scheme says what to look for. It never says "a point if it seems good".

A harness works the same way:

1. The builder does not mark its own work.
2. The marker uses a list of lines it can check.

Click the button and open the Sessions & App tab.

• The verdicts: the builder passes all six samples.
• The reason: each time it says "I wrote this, and it looks right to me."
• The truth: only one sample was really done.

Simple lesson: a builder that grades itself passes almost everything.

What You Will Learn

In this lesson, you will learn:

1. Why self-review passes bad work.
2. The planner, generator and evaluator roles.
3. How to write a rubric that the evaluator can check.
4. How to calibrate an evaluator on known samples.
5. How the three roles fit UI design, frontend, backend and integration work.
6. How to review the agent's work yourself before you merge it.

Why Self-Review Fails

The Builder Reads Kindly

Self-review = the agent that built the work also judges it.

The builder has a story in its context: "I added the Export button and turned it on."

When it reviews the code, it reads through that story.

It sees what it meant to do. It does not see what is really there.

Anthropic's team saw this in their work on long-running apps.

An agent that judged its own output praised it. It did so even when a person could see the work was weak.

Step through all four frames.

• Frame 1: six samples. A person already judged each one.
• Frame 2: the builder passes all six. It agrees with the person only once.
• Frame 3: a separate evaluator reads the same samples with a rubric. Each FAIL has a reason.
• Frame 4: both judges side by side, checked against the person.

A Vague Rubric Is No Better

Rubric = the evaluator's checklist. Each line is one thing to check.

A separate evaluator does not fix things by itself. It also needs to know what to check.

The starting rubric in the lab says:

- The code looks good.
- The feature seems done.

"Looks good" to whom? "Seems done" by which test?

Neither line names anything a checker can see. So the evaluator has nothing to fail the work on.

Simple lesson: a separate judge with a vague list is still a kind judge.

Three Roles: Planner, Generator, Evaluator

The Planner

Planner = the agent that turns a short goal into a full plan.

In this course, the plan is feature_list.json.

It lists every feature. Each one has a description and "passes": false.

The planner can also write design notes, like which colour tokens the UI uses.

The planner does not write code. Its job is to make the work clear first.

The Generator

Generator = the builder.

It picks the next feature where passes is false. It builds it, runs the tests and commits.

This is the agent from the Incremental Sessions lesson. One feature per session, with a clean exit.

The generator does not decide that the feature is done. It asks the evaluator.

The Evaluator

The evaluator starts with its own fresh context. It did not write the code, so it has no story to protect.

It reads the rubric and runs or reads the checks.

Then it answers PASS or FAIL with a reason.

If the answer is FAIL, the reason goes back to the generator. The generator fixes the problem and asks again.

Simple lesson: only the evaluator's PASS counts, and its FAIL always has a reason.

Write and Calibrate the Rubric

One Line for Each Thing a Checker Can See

List the ways work can look done but not be done. Write one line for each:

Here is the full file:

# Review rubric

You are the evaluator. You did not write this code. Answer PASS or FAIL.
FAIL if any line below is false.

- The unit tests pass.
- The end-to-end check passes in the running app.
- No test was skipped or switched off.
- The feature description in feature_list.json is unchanged.
- Only files for this feature changed; no unrelated edits.

For the React Taskboard, unit tests means pnpm test. End-to-end means pnpm e2e.

For the ASP.NET Core Pricing API, they mean dotnet test and dotnet test --filter Category=E2E.

Step through all three frames.

• Frame 1: the vague rubric. Both lines are marked "checks nothing".
• Frame 2: the full rubric. Each line shows which sample it catches.
• Frame 3: the evaluator's verdicts, each with its reason.

Then the rules file names the evaluator step:

- When a feature is built, ask the evaluator to review it with evaluator/rubric.md.
- Flip "passes" to true only after the evaluator says PASS.

Calibrate on Known Samples

Calibrate = test the evaluator on cases where you already know the right answer.

You keep a small set of samples.

Some are good. Each bad one has one known problem.

Run the evaluator on them. Count how often it agrees with the person.

Step through all five frames.

• Start: the vague rubric agrees on only one sample.
• Each new line: the count goes up. The frame names the sample it fixed.
• The end: all lines together agree on every sample.

Lab Exercise: Calibrate a Separate Evaluator

Open evaluator/rubric.md in the lab. Replace the two vague lines.

Write one line each for unit tests, the end-to-end check, skipped tests, the description and unrelated edits.

Click the button, edit the rubric, then press Check my harness.

• The check: the evaluator runs your rubric on the six samples.
• To pass: it must judge all 6 correctly.
• If it misses one: read that sample and ask which line would catch it.

Simple lesson: each miss points to a missing line.

The Three Roles in a Team Workflow

UI Design Review

The rubric idea also works for design. Anthropic's team graded designs on quality, originality, craft and how well they work.

To make lines like these checkable, tie each one to something a reviewer can see:

- Every colour comes from src/styles/tokens.ts (lint passes).
- Every button has a name (the accessibility check passes).
- The page works at 375px wide with no sideways scroll (screenshot check).

Planning and Task Scheduling

The planner writes feature_list.json.

In a team, an owner field (see Feature Lists) lets people or agents split the work.

The evaluator reviews each item before it counts as done. It does not matter who built it.

Frontend and Backend

For the React Taskboard, the generator builds src/features/<id>.ts and turns it on in src/App.tsx.

The evaluator checks pnpm test and pnpm e2e. Playwright clicks through the page like a user.

For the Pricing API, the generator builds Pricing.Api/Features/<Id>.cs and maps it in Program.cs.

The evaluator checks dotnet test and dotnet test --filter Category=E2E. These make HTTP calls to the running API.

Testing and Integration

The evaluator is the last gate before passes flips to true.

CI (see Tests and Linters) runs every check on a clean machine for every change.

The evaluator does not replace CI. It adds checks that a command cannot do alone:

• Was the description changed?
• Were unrelated files touched?
• Was a test switched off?

Simple lesson: only an evaluator PASS may flip passes to true.

You Are the Last Evaluator

An evaluator agent helps. But a person still decides what gets merged.

Before you merge the agent's work, review it like a teammate's pull request:

1. Read the whole diff. If it is too big to read, ask for a smaller change.
2. Check which files changed. Look for files you did not expect, protected files and test files.
3. Look for weakened tests. A test switched off, deleted or changed to expect the wrong answer.
4. Run it yourself. Run the tests, and try the feature in the running app.
5. Compare the notes with the diff. The feature list and progress.md should match what really changed.

Click the button, let the run finish, then open the Review tab.

• Every changed file is listed with its diff.
• Flag each change you would not merge. Then choose Merge or Request changes.
• The check tells you what you missed.

Anything you catch becomes one rubric line or one check. Continuous Improvement shows how.

An evaluator is a second agent run, so it costs time and money. Use one for long or risky work, not for every small change.

Simple lesson: read the diff before you merge. Turn each catch into a check.

TL;DR

1. Self-review = the builder judges its own work. It passes almost everything.
2. Planner = turns the goal into feature_list.json.
3. Generator = builds one feature, then asks for review.
4. Evaluator = a separate agent that answers PASS or FAIL with a reason.
5. Rubric = the evaluator's checklist. Every line is something a checker can see.
6. Calibrate = test the rubric on known samples until it agrees with a person.

Common mistakes

Check Your Understanding

Exercises

These exercises practise the three roles and the rubric.

Press Check when you are ready. Use Show Hint if you get stuck.

What to Remember

• The builder should not judge its own work. Self-review passes almost everything.
• Use three roles. The planner plans, the generator builds, and a separate evaluator judges.
• Every rubric line must be something a checker can see.
• Calibrate the evaluator on known good and bad samples.
• Add a new sample whenever something slips through.
• Only an evaluator PASS may flip passes to true.

Further Reading

• Harness design for long-running application development, by Anthropic: the planner, generator and evaluator agents, and why self-evaluation is too kind.
• Effective harnesses for long-running agents, by Anthropic: the feature list, the progress file and one feature per session, which the generator follows.
• Harness engineering for coding agent users, on martinfowler.com: checks and feedback that the agent can act on.

Next: Subagents and MCP. Part 3 starts by keeping the main agent's context clean as the project grows.