Cost, Latency and SLOs

By the Syncfusion DevTutorials team · Last updated October 27, 2026

The team switched to a bigger model. Quality went up, and so did the bill, by seven times. Learn to measure cost per request, p50 and p95 latency, error and refusal rates, set SLOs, and find the step that blows the budget.

Cost, Latency and SLOs

The team switched Shop Helper to the bigger model. That is v4.

The eval said v4 was better. It was. The team shipped it and moved on.

A month later, the finance team called.

One chat now costs 0.708¢ instead of 0.1¢. At 300,000 chats a month, that is about $2,123 instead of $301.

Nobody was watching the cost. Nobody was watching the wait either.

Simple lesson: quality is one number. Cost and speed are two more, and they need watching too.

A Real-World Example

Think about a taxi app.

A ride can be safe and pleasant. But if it costs three times the normal fare, or the car takes 25 minutes to arrive, people stop using the app.

So the app watches three things on every ride: the quality, the price and the wait.

Your AI feature is the same. Every chat has a quality, a cost and a wait.

Simple lesson: watch quality, cost and speed together.

What You Will Learn

In this lesson, you will learn:

1. How to work out cost per request from tokens and prices.
2. What p50 and p95 latency are, and why the average misleads.
3. How to find the step in a trace that blows the budget.
4. What an SLO is, and how to set one.
5. Which number to watch for slow replies, high bills, outages and refusals.

TL;DR

Cost per request = input tokens × input price + output tokens × output price, added up over every model call.

Latency = how long a reply takes.

p95 = the time that 95% of requests beat. 1 in 20 is slower. Watch p95, not the average.

SLO (service level objective) = a line you promise not to cross, such as "daily p95 under 3000 ms".

Set SLOs so normal days pass and bad days fail. Then a broken SLO means a real problem.

Cheat Sheet

Cost per Request

Models charge by the token. Input tokens and output tokens have different prices. Output tokens usually cost more.

Token = a small piece of text. Models read and write tokens, and you pay for both.

Here is the sum for one chat.

Cost of one chat, and of a month

// Price per 1,000 tokens, in cents. Output tokens cost more than input.
const prices: Record<string, { input: number; output: number }> = {
  "model-s": { input: 0.15, output: 0.6 },
  "model-l": { input: 1.0, output: 4.0 },
};

function costCents(model: string, inputTokens: number, outputTokens: number): number {
  const p = prices[model];
  return (inputTokens * p.input + outputTokens * p.output) / 1000;
}

const perChat = costCents("model-l", 560, 30);
console.log(`one chat: ${perChat.toFixed(3)} cents`);
console.log(`300,000 chats: $${((perChat * 300000) / 100).toFixed(0)}`);
console.log(`same on model-s: $${((costCents("model-s", 560, 30) * 300000) / 100).toFixed(0)}`);

Expected output:

one chat: 0.680 cents
300,000 chats: $2040
same on model-s: $306

A fraction of a cent per chat looks like nothing. Multiplied by a month of traffic, it is a real bill.

These prices are made up for the course. Real prices change often, so read them from your provider's price page.

Simple lesson: cost per request × requests per month = the bill. Watch both parts.

Latency: Average, p50 and p95

Most chats are quick. A few are slow. The slow ones are the ones people complain about.

For v1, the average is 1116 ms, p50 is 1121 ms and p95 is 1462 ms. v4, the bigger model, has a p95 of 2879 ms.

p50 = the middle time. Half the requests are faster.

Percentile = the value that a given share of requests beat. p95 means 95%.

The average hides the slow tail. One very slow chat pulls the average up, but it does not tell you how many people waited.

Average against p50 and p95

// Reply times in ms for 10 chats.
const times = [940, 1210, 1050, 980, 4300, 1120, 1005, 1180, 990, 1075];

// Nearest-rank percentile: sort, then take value number ceil(p% of n).
function percentile(xs: number[], p: number): number {
  const sorted = [...xs].sort((a, b) => a - b);
  const rank = Math.ceil((p / 100) * sorted.length);
  return sorted[rank - 1];
}

const average = times.reduce((a, b) => a + b, 0) / times.length;
console.log(`average ${average} ms`);
console.log(`p50 ${percentile(times, 50)} ms`);
console.log(`p95 ${percentile(times, 95)} ms`);

Expected output:

average 1385 ms
p50 1050 ms
p95 4300 ms

The average says 1385 ms. Nobody waited 1385 ms. Most waited about one second, and one person waited over four.

Try this: change 4300 to 1300. Predict the three numbers, then run it.

Simple lesson: report p50 for the typical wait and p95 for the slow tail.

Find the Step That Blows the Budget

Say the budget is 0.5¢ per chat. A v4 chat costs 0.687¢. Which step is to blame?

The trace knows. Every model-call span has its token counts.

The first model call only picks a tool. That is a simple job, but it runs on the big model.

Move that call to the small model, and keep the big model for the reply. The chat drops to about 0.395¢, under budget.

Simple lesson: price each span. Spend the big model where it matters.

SLOs: Lines You Promise Not to Cross

A dashboard full of numbers is hard to watch. An SLO turns each number into a yes or no.

SLO (service level objective) = a target for a number over a time window, such as "daily p95 under 3000 ms".

Here are the 14 days from the dashboard. Each row is the day's average of hourly p95, and the average cost per chat.

Day 8 had a latency regression. It went on into the first hours of day 9. Day 6 had a cost spike. All other days were normal for these two numbers.

A good SLO sits above the normal days and below the bad ones. Too low, and it pages people every day. Too high, and it misses real problems.

Simple lesson: an SLO should be quiet on normal days and loud on bad ones.

Errors and Refusals

Two more numbers belong on the dashboard.

Error rate = the share of requests that failed, for example because the model provider was down.

Refusal rate = the share of replies where the helper refused to help, such as "I can't help with that".

A refusal is not an error. The call worked, and the reply was useless. Count it on its own, with a simple check on the reply text, or with a judge.

Where It Is Used

• Budget planning: cost per request × expected traffic.
• Model choice: is the better model worth its cost and its wait?
• On-call: SLOs decide when someone gets paged.
• Prompt reviews: a longer prompt is a cost on every single chat.

Common mistakes

Check Your Understanding

Exercises

What to Remember

• Every chat has a quality, a cost and a wait. Watch all three.
• Cost per request = tokens × price, added over every model call.
• Watch p95 latency. The average hides the slow tail.
• Price each span to find the step that blows the budget.
• An SLO is a line you promise not to cross. Normal days meet it, bad days break it.
• Track errors and refusals as separate numbers.

Next: Online Evals and Monitoring, how to score live traffic and catch a quality drift the day it starts.