Online Evals and Monitoring

By the Syncfusion DevTutorials team · Last updated October 27, 2026

Quality starts to drift on day 4 and nobody notices for three days. Learn to judge a sample of live chats, read user feedback, and tune an alert that catches the drift within a day without crying wolf.

Online Evals and Monitoring

Shop Helper passed its offline eval. The team shipped it on a Monday.

On day 4, the model provider pushes an update. Replies get a little worse each day.

Nothing crashes. No error shows up. The cost chart looks normal.

On day 7, customers start to complain in reviews. The team finds out from angry posts.

Watch the week.

The team did test before shipping. They just stopped measuring after shipping.

Simple lesson: an offline eval checks the release. An online eval checks every day after it.

A Real-World Example

Think about a bakery.

Before a new bread goes on sale, the baker tastes a test loaf. That is the offline eval.

But flour changes. Ovens drift. A new helper shapes the dough a bit differently.

So a good bakery also tastes one loaf from every batch, every morning. That is the online eval.

They do not taste every loaf. One from each batch is enough to notice a problem early.

Simple lesson: taste a sample every day, not just once at the start.

What You Will Learn

In this lesson, you will learn:

1. What an online eval is, and how it differs from an offline eval.
2. How much live traffic to judge, and what that costs.
3. How to use thumbs-down feedback as a second signal.
4. How to set an alert that catches a drift within a day.
5. How to avoid alert fatigue.

TL;DR

Online eval = scoring a sample of real production chats, all the time, with the same graders and judge you used offline.

Drift = quality that slides slowly over days, with no crash and no error.

Judge a sample of live chats every hour. Alert on the average score over a few hours. Give every alert an owner and a runbook.

Too small a sample makes a noisy score. A noisy score makes false alerts. False alerts get muted.

Cheat Sheet

The same ideas, with the names your platform uses.

Offline and Online Evals

You already know offline evals. You run a fixed set of saved cases before you ship.

An online eval runs on live chats, after you ship.

It answers a different question. Not "is this version good?" but "is it still good today?"

Three things change from offline to online:

• No answer key. Live chats have no expected answer. So you need graders that work without one: code checks for rules, and an LLM judge for quality.
• Too many chats. You cannot judge every chat cheaply. So you judge a sample.
• It runs forever. So the result must be a number over time, with an alert on it.

Sample = a share of live chats picked at random to be scored, such as 1 chat in 10.

Simple lesson: online evals use the same graders as offline, on a sample of real traffic, every hour.

How Much Traffic to Judge

Each judge call costs money. Judging every chat gives a steady score and a big bill.

Judging very few chats is cheap. But then each hour's score depends on a handful of chats, so it jumps around.

The lab tested six sampling rates with the same alert rule. The bar: catch the drift within 24 hours, with 2 false alerts or fewer in 14 days.

The cheapest rate that meets the bar is 10%. Judging 1 chat in 10 costs about $24 for 14 days. Judging everything would cost about $244.

Simple lesson: judge enough chats to make the score steady, and no more.

Thumbs Down Is a Second Signal

Many apps have a thumbs-up and thumbs-down button. That is free feedback from real users.

User feedback signal = something users do that tells you how a reply went, such as a thumbs down, a retry, or asking for a person.

It is real, but it has two problems. Most people never vote. And angry people vote more than happy ones.

So do not use thumbs down as your main alarm. Use it to find chats to read, and to check that the judge still agrees with real users.

A thumbs-down chat is a ready-made test case. Add it to your eval set. That is step 8 of the eval loop: feed back.

Simple lesson: alert on the judge. Learn from the thumbs.

Alerts That People Trust

An alert rule watches a number and pages someone when it crosses a line.

The team's first rule was: "quality below 88% in any single hour". It sounds careful.

In 14 days, the first rule fired 23 alerts, and 13 of them were false.

Alert fatigue = people stop reacting to alerts because too many of them were false.

A muted alert is worse than no alert. It makes the team feel safe while nobody is watching.

The fix is an average over a window. A 6-hour average below 87.5% catches the drift after 19 hours, with no false alerts.

Window = how many hours of scores the rule averages before it compares with the line.

Simple lesson: a good alert is quiet when things are fine and loud when they are not.

Try the Window Yourself

Here is one day of hourly judge scores. A drift starts at hour 12.

The cell runs the same rule two ways: with a 1-hour window and with a 6-hour window.

1-hour window vs 6-hour window

// Hourly judge scores (% of sampled chats that passed) for one day.
// A drift starts at hour 12 and slowly pulls the score down.
const scores = [91, 88, 93, 90, 87, 92, 91, 89, 93, 90, 92, 91,
  89, 88, 89, 87, 87, 86, 86, 85, 86, 84, 85, 84];

function alertHours(windowHours: number, line: number): number[] {
  const fired: number[] = [];
  let on = false;
  for (let h = windowHours - 1; h < scores.length; h++) {
    const win = scores.slice(h - windowHours + 1, h + 1);
    const avg = win.reduce((s, x) => s + x, 0) / win.length;
    const hit = avg < line;
    if (hit && !on) fired.push(h);
    on = hit;
  }
  return fired;
}

console.log("1-hour window:", alertHours(1, 88).join(", "));
console.log("6-hour window:", alertHours(6, 88).join(", "));

Expected output:

1-hour window: 4, 15
6-hour window: 17

The 1-hour rule fires at hour 4. That is before the drift, so it is a false alert. One noisy hour was enough.

The 6-hour rule fires once, at hour 17. It is a little later, and it is real.

Try this: change the 6-hour window to 12. Predict what happens, then run it. A window that is too long can miss a drift that started late in the day.

Every Alert Needs an Owner

An alert that pages nobody does nothing.

Runbook = a short page that says what to check and what to do when an alert fires.

Each rule should say:

• Who is paged (a person or a team).
• Where the runbook is.
• What "fixed" looks like, so the alert can close.

The next lesson follows an alert from the page to the root cause.

Where It Is Used

• After every release, to catch problems the eval set did not cover.
• When a provider updates a model you did not change.
• When your data changes, such as new help pages or new products.
• When attackers try prompt injection on live traffic.

Common mistakes

Check Your Understanding

Exercises

What to Remember

• An offline eval checks the release. An online eval checks every day after it.
• Judge a random sample of live chats with the same graders and judge.
• Too small a sample makes a noisy score. Pick the smallest sample that stays steady.
• Use thumbs down to find chats to read, not as the main alarm.
• Alert on an average over a window, and give every alert an owner and a runbook.

Next: Debugging from Traces, how to follow an alert to the trace, find the root cause and add a regression case.