Where should the cost test run?

In CI, on every prompt or model change

Measure in Production

By Syncfusion®, Inc. · Last updated November 2, 2026

Normalise token usage from OpenAI-style and Anthropic-style APIs, attribute LLM cost and p95 latency by feature, tenant and prompt version, and write a cost test that fails CI when a prompt change breaks a budget, in TypeScript or C#.

What you will learn

Normalise usage blocks from OpenAI-style and Anthropic-style APIs, including cache and reasoning tokens
Attribute cost, share and p95 latency by feature, tenant and prompt version
Write a CI cost test that fails when a change breaks a cost or latency budget

One Total Is Not Enough

The monthly AI bill went up 40%. Your manager asks why. Which feature? Which customer? Which change?

If your logs only have a total, you cannot answer. You need every call tagged, and token counts you can trust from every provider.

Press Play, or use Next to go one step at a time.

Frame 1: 60 calls, one total.
Frame 2: the same calls by feature.
Frame 3: prompt v13 against v12, and the CI check.

Simple lesson: tag every call, so every question about cost has an answer.

A Real-World Example

A team added worked examples to their support prompt. The replies got a little better. Nobody looked at cost.

Three weeks later, the bill was almost double. The new prompt version had doubled input tokens, on the most used feature.

Simple lesson: prompt changes are cost changes, so test them like code.

What You Will Build

In this lesson, you will learn:

how to read usage from different providers into one shape
how to attribute cost by feature, tenant and prompt version
how to fail CI when a change breaks a budget

You will finish three files in the telemetry folder:

Usage (src/telemetry/usage.ts, or Telemetry/UsageNormalizer.cs): one token count for every provider.
Attribution (src/telemetry/attribution.ts, or Telemetry/Attribution.cs): cost by tag.
Cost test (src/telemetry/costTest.ts, or Telemetry/CostTest.cs): the CI check.

Open the Code tab. The harness has usage blocks in two provider shapes, 60 tagged production records and a budget per feature.

Simple lesson: the meter from Lesson 2 is the source, and tags make it useful.

One Usage Shape for Every Provider

Providers report tokens differently:

OpenAI-style APIs report prompt_tokens including cached tokens, and put reasoning tokens inside completion_tokens
Anthropic-style APIs report input_tokens without the cache, and cache reads and writes in their own fields
some streams send no usage at all

If you add the fields the same way for both, one provider looks much cheaper than it is.

Run the cell. The same call reads the same after normalising, whichever provider it came from.

Missing usage is not free. Flag it and count it. A stream cut off before the last chunk often has no usage block. Estimate it, or count the call as unknown.

Simple lesson: normalise once, at the edge, then everything downstream is simple.

Cost by Tag

Every record carries three tags:

feature: support reply, ticket tagging, help search
tenant: the customer account
prompt version: the version of the prompt that made the call

Group by any tag, and report requests, dollars, share and p95.

Run the cell. Support replies are 94% of the spend. Try "tenant" and "promptVersion".

OpenTelemetry has conventions for AI calls. The GenAI semantic conventions name attributes for the model and the token counts, such as gen_ai.request.model and gen_ai.usage.input_tokens. They are still evolving, so check the current spec. Add your own attributes for feature, tenant and prompt version.

Simple lesson: attribution tells you where to optimise, and who to talk to.

A Cost Test in CI

The eval traffic from Lesson 1 runs on every prompt change. Add one more check: for each feature, the average cost per request and the p95 must stay inside a budget.

Run npm run cost-test (or dotnet run -- cost-test). Prompt v12 passes. Prompt v13 fails: support replies cost $0.00616 per request, over the $0.004 budget.

Try it on a real model. Read the real usage block from a model in your browser, and see what your meter would log. Switch Model to Real in the lab's top bar. It runs an open model in your browser with WebLLM, and needs WebGPU. It is not graded.

Simple lesson: a prompt change that breaks a budget should fail the build.

Put It in Your App

Tag at the call site. The wrapper that meters calls takes feature, tenant and prompt version as required arguments.
Send records to one place. A table in your warehouse, or spans in your tracing tool, with the tags as columns.
Run the cost test in CI. Replay the golden set for every prompt or model change, and exit non-zero on a failure.
Build one dashboard. Cost per feature per day, p95 per feature, and the share of calls with missing usage.

Sample, do not drop. Keep every cost record, even if you sample traces. Cost totals must add up to the bill, and sampled data never does.

For product managers: ask for cost per feature and per customer every month. It shows which features pay for themselves, and which customers need a different plan.

Simple lesson: tags, one store, one CI check and one dashboard.

Common mistakes

Mistake	Wrong idea	Correct idea	What goes wrong
Adding usage fields the same way	"Tokens are tokens."	Normalise per provider.	Cache tokens are double counted or lost.
Treating missing usage as zero	"No usage, no cost."	Flag and count it.	Spend hides in cut-off streams.
No prompt version tag	"We know what changed."	Tag every call with it.	You cannot find the costly change.
Cost only in the monthly bill	"Finance will tell us."	Test cost in CI.	You find out weeks later.
Checking the cheapest request	"Most calls are fine."	Check the average and p95.	Expensive calls slip through.

Check Your Understanding

Exercises

Build It in the Project

The other files of this lesson's project are optional practice. Finish them to run the lesson's script on your own code.

What to Remember

Providers report tokens differently, so normalise usage first.
Missing usage is flagged, not counted as free.
Tag every call with feature, tenant and prompt version.
Attribute cost and p95 by tag to know where to optimise.
A cost test in CI fails the build when a budget breaks.

Next: The Token Diet. You will cut the input tokens that make most of the bill, and check the answers with the quality gate.