Evals in CI + v1 Export

By the Syncfusion DevTutorials team · Last updated October 27, 2026

A friendlier prompt merged with two approvals and no eval. Build a CI gate that runs the eval set on every prompt, model or retrieval change, blocks the regression, lets fine changes through, and fits the time budget. Then export your v1 eval suite.

Evals in CI + v1 Export

Remember v2, the friendlier prompt from the first lesson?

Here is how it got shipped. It was a pull request that changed one file.

Every check was green. Two people approved it. The eval set existed, but nobody ran it.

A prompt change is a code change. It needs a test that looks at what the feature says and does.

Simple lesson: if a change can alter the AI's answers, the eval set must run before it merges.

A Real-World Example

Think about a bakery that changes its bread recipe.

The new recipe looks fine on paper. The head baker approves it.

But a good bakery bakes a test batch first. It checks taste, size and the salt level every time, before the bread goes on the shelf.

The test batch is the eval. The rule "no new recipe without a test batch" is the gate.

Simple lesson: a gate turns "we should test" into "we always test".

What You Will Learn

In this lesson, you will learn:

1. What CI and an eval gate are.
2. Which file changes should start the eval run.
3. How to set a gate that blocks a regression hidden inside a good total.
4. Why a gate must also let fine changes through.
5. How to check that the eval set fits the CI time budget.
6. How to export your v1 eval suite: the files you take home.

TL;DR

CI (continuous integration) = checks that run by themselves on every pull request.

Eval gate = a CI step that runs the eval set and blocks the merge if the result is not good enough.

Run it when prompts, model settings, retrieval or tools change.

Gate on two things: the total pass rate, and how far each kind of chat dropped. The total alone can hide a broken kind.

Cheat Sheet

The Gate in Action

Here is the same pull request, with an eval gate in CI.

The gate compares the pull request's version with main, the version that is live now.

Baseline = the score of the version you compare against. Here it is main (v1).

It looks at each kind of chat. Refunds dropped by 42 points. The gate blocks the merge and posts why.

The author never had to remember to run the eval. CI did it.

Simple lesson: compare with main, kind by kind, and block when something important dropped.

Total-Only Gates Miss Things

A common first gate checks one number: "the pass rate must be at least 80%".

v2 scores 82%. That gate lets it through.

A per-kind rule fixes it. "No kind of chat may drop more than 5 points" blocks v2.

But a gate must also let good changes through. v3 is a small, neutral fix. Its worst kind moves by 3.3 points, which is noise-sized. v4 is better everywhere. Both must pass.

False alarm = a gate that blocks a change that was fine.

A gate full of false alarms gets switched off. Then it catches nothing.

Here is the gate rule in code. It checks the total and every kind.

A total-only gate against a per-kind gate

type Kind = { name: string; main: number; pr: number };
type Gate = { minRate: number; maxDrop: number };

function check(gate: Gate, total: number, kinds: Kind[]): string {
  const problems: string[] = [];
  if (total < gate.minRate) problems.push(`total ${total}% < ${gate.minRate}%`);
  for (const k of kinds) {
    const drop = k.main - k.pr;
    if (drop > gate.maxDrop) problems.push(`${k.name} -${drop} points`);
  }
  return problems.length ? `BLOCK (${problems.join(", ")})` : "PASS";
}

// PR 41: the friendlier prompt. Pass rates per kind, main vs PR.
const pr41: Kind[] = [
  { name: "order status", main: 93, pr: 93 },
  { name: "refunds", main: 92, pr: 50 },
  { name: "address", main: 87, pr: 87 },
];

const totalOnly: Gate = { minRate: 80, maxDrop: 100 };
const perKind: Gate = { minRate: 80, maxDrop: 5 };

console.log("total only:", check(totalOnly, 82, pr41));
console.log("per kind:  ", check(perKind, 82, pr41));

Expected output:

total only: PASS
per kind:   BLOCK (refunds -42 points)

Try this: change the refunds pr value to 89. Predict both lines, then run it.

When to Run the Gate

Running 200 model calls on every pull request costs time and money. Run them only when the AI's behaviour can change:

• Prompts, such as prompts/shop-helper.md.
• Model settings, such as config/model.json.
• Retrieval, such as how help pages are searched.
• Tool code, such as the refund tool.

A change to the README or a button colour cannot change an answer. Skip the eval for those.

Fit the Time Budget

CI has a time limit. Say yours is 5 minutes, and CI can run 4 cases at once.

One Shop Helper case takes about 1.1 seconds. So about 1,075 cases fit in the budget.

The 200-case eval set fits with lots of room. That room is for the cases you will add later.

If the set grows too big, run a fast subset on every pull request, and the full set every night.

Export Your v1 Eval Suite

This is the end of the first part of the course. You have a taxonomy, an eval set, code graders, a judge rubric and a gate.

The Export tab in the Eval Lab packs them into files you can add to a real repo.

Open .github/workflows/evals.yml in the export. Two parts matter:

• The paths list is the trigger. It names the AI files.
• The last step is the gate. It fails the job when the pass rate is too low, or a kind drops too far.

Where It Is Used

• Every pull request that touches prompts, model settings, retrieval or tools.
• Model upgrades, even "minor" ones from the provider.
• Nightly runs of the full set, when the per-PR set is a subset.

Common mistakes

Check Your Understanding

Exercises

What to Remember

• A prompt change is a code change. It needs the eval set, not just a review.
• An eval gate runs the eval set in CI and blocks bad changes.
• Trigger it on prompts, model settings, retrieval and tools.
• Gate on the total and on how far each kind dropped.
• Set limits above the noise, so fine changes pass.
• Keep the per-PR run inside the CI time budget.
• Export your v1 eval suite and add it to your repo.

Next: Tracing with OTel GenAI Conventions, how to see every step your feature takes in production.