Experiments for Prompts and Models

By the Syncfusion DevTutorials team · Last updated October 27, 2026

A bigger model wins the demo. Before anyone switches, learn to version prompts, shadow-test a new model on live traffic, weigh quality against cost and latency, and roll a change out as a canary behind the CI gate.

Experiments for Prompts and Models

A new, bigger model comes out. Someone on the team tries it on five chats. The replies are great.

"Let's switch everyone tonight," they say. "It is clearly better."

It is better. It also costs about seven times as much per chat. And it is twice as slow.

Nobody checked, because the demo only showed quality. This lesson shows how to test a change on real traffic first. You get numbers for quality, cost and speed before customers see it.

After 500 live chats, the bigger model (v4) passes 96.8% and v1 passes 91.6%. That gain is real.

But v4 costs 0.715¢ per chat against 0.1¢. Its p95 latency is 2922 ms against 1451 ms.

Simple lesson: a better answer is not always worth its price.

A Real-World Example

Think about a restaurant trying a new supplier.

They do not switch the whole menu overnight. First the chef cooks the new ingredient in the back. They taste it next to the old one. Guests never see it.

Then one table a night gets the new dish. If nobody complains, and the cost works out, more tables get it.

That is a shadow test, then a canary.

Simple lesson: try it in the back first, then on a few tables, then on everyone.

What You Will Learn

In this lesson, you will learn:

1. Why prompts need version ids, like code.
2. What a shadow test is, and why it is safe.
3. How to decide: promote, reject, or run longer.
4. How to weigh quality against cost and latency.
5. How to roll a change out as a canary behind the CI gate.

TL;DR

Experiment = a planned test that compares a new version with the current one on the same traffic, with a decision rule written down first.

Shadow test = the new version answers a copy of live chats in the background. Users only see the current version.

Canary = a small share of real users gets the new version first, while alerts compare them with everyone else.

Promote a change only when it is better beyond the noise, inside the cost budget, and inside the latency target. Then roll it out in steps.

Cheat Sheet

Prompts Are Versions

A prompt changes your app as much as code does. So treat it like code.

Prompt version = a saved, numbered copy of a prompt, such as shop-helper.v3. It lives in a file, in git, and changes only through a pull request.

Two things make versions useful:

• Every trace records which version answered (app.version in the spans). So any reply can be traced to its prompt.
• The CI gate from lesson 7 runs on any pull request that changes a prompt file. So no prompt reaches users without passing the eval set.

Simple lesson: no prompt change without a version id and a pull request.

The Shadow Test

The eval set is good, but it is fixed. Live traffic has new kinds of chats every week.

A shadow test runs the new version on real chats, with no risk. The customer still gets v1's reply. The new reply is only saved and scored.

It costs money. You pay for two model calls per shadowed chat. So shadow a sample, not everything. And stop when you have enough.

Because both versions answer the same chats, you can compare them paired, like in lesson 6. That keeps the noise small.

Decide With Three Numbers

Write the decision rule down before you look at the results. Otherwise it is easy to argue for the version you hoped for.

This team's rule:

• Quality must be better, and the 95% interval must be above 0.
• Cost must stay under 0.15¢ per chat.
• p95 latency must stay under 2500 ms.

This cell applies the rule to three shadow tests.

Promote, reject or run longer

// Shadow-test results: each candidate against v1 on the same live chats.
type Result = { name: string; gain: number; low: number; high: number; cents: number; p95: number };

const BUDGET_CENTS = 0.15;
const P95_TARGET_MS = 2500;

const results: Result[] = [
  { name: "v2 prompt", gain: -4.2, low: -8.2, high: -0.8, cents: 0.108, p95: 1441 },
  { name: "v3 prompt", gain: 3.8, low: 0.6, high: 7, cents: 0.105, p95: 1429 },
  { name: "v4 model", gain: 5.2, low: 2.4, high: 8, cents: 0.715, p95: 2922 },
];

function decide(r: Result): string {
  if (r.high < 0) return "reject: worse";
  if (r.cents > BUDGET_CENTS) return "reject: over budget";
  if (r.p95 > P95_TARGET_MS) return "reject: too slow";
  if (r.low <= 0) return "run longer: could be noise";
  return "start a canary";
}

for (const r of results) console.log(`${r.name.padEnd(10)} ${decide(r)}`);

Expected output:

v2 prompt  reject: worse
v3 prompt  start a canary
v4 model   reject: over budget

The small prompt fix (v3) passes 95.4% on the same chats. That is most of v4's gain, at almost no extra cost.

Try this: raise BUDGET_CENTS to 1. Run it again. v4 is still rejected. Which rule stops it now?

The Canary Rollout

A shadow test cannot tell you how users react. A canary can.

Roll the change out in steps. At each step, alerts compare the canary group with everyone else. If anything looks wrong, switch back in one click.

Rollback switch = a setting that sends everyone back to the old version at once, with no new deploy.

The canary never skips the earlier steps. A change that fails the CI gate never reaches a shadow. A change that fails the shadow never reaches a customer.

Simple lesson: each step is a gate, and each gate can stop the rollout.

Where It Is Used

• A new model from your provider, or a new provider.
• A prompt change that touches many kinds of chat.
• A retrieval change, such as a new search setting or new chunk size.
• A cost cut, such as a smaller model for some chats. A shadow test shows what quality you would lose.

Common mistakes

Check Your Understanding

Exercises

What to Remember

• Keep prompts in versioned files, and record the version on every trace.
• A shadow test scores a new version on live chats with no risk to users.
• Decide with three numbers: quality beyond the noise, cost, and latency.
• Write the decision rule before you look at the results.
• Roll out as a canary in steps, behind the CI gate, with a rollback switch.

Next: Capstone: LLMOps Pack for Your Feature, where you put every piece together and test it against six real incidents.