Latency Engineering

By Syncfusion®, Inc. · Last updated November 2, 2026

Cut the time users wait: schedule model calls as a graph, measure p50, p95 and p99 against an SLO, and use streaming, a non-blocking judge and an acknowledgement, in TypeScript or C#.

What you will learn

• Schedule model calls as a dependency graph, in a row or in parallel
• Measure p50, p95 and p99 and check them against an SLO
• Find when the first visible text appears with streaming, a judge and an acknowledgement

Eleven Seconds Before the First Word

The product promise for the order agent is simple: the user sees an answer starting within 4 seconds, 95% of the time. That is a latency SLO (service level objective).

Today, the median ticket takes 11.5 seconds and the p95 is 17.7 seconds. All 40 recorded tickets break the SLO.

The code is the problem as much as the models. It awaits every step in turn, and a safety judge holds the reply until the whole answer is checked.

Press Play, or use Next to go one step at a time.

• Frame 1: today: every step waits for the one before it.
• Frame 2: each step starts when its inputs are ready.
• Frame 3: the reply streams, and the judge checks afterwards.

Simple lesson: the user waits for the critical path, and feels the first visible text most.

A Real-World Example

The profile lookup and the help search did not need anything from each other. They still ran one after another, because the handler was written as a list of await lines.

Nobody noticed, because each step was fast on its own.

Simple lesson: draw the calls as a graph, then run what can run together.

What You Will Build

In this lesson, you will learn:

1. how to schedule calls as a graph, in a row or in parallel
2. how to measure p50, p95 and p99, and check them against an SLO
3. how streaming, a non-blocking judge and an acknowledgement change what the user feels

You will finish three files in the latency folder:

• Schedule (src/latency/schedule.ts, or Latency/Schedule.cs): finish times and the total.
• Percentiles (src/latency/percentiles.ts, or Latency/Percentiles.cs): p50, p95, p99 and breaches.
• Perceived (src/latency/perceived.ts, or Latency/Perceived.cs): when the first text appears.

Open the Code tab. The harness has the ticket as steps with their recorded times, their time to first token, and the steps each one needs first.

Simple lesson: latency work starts with a dependency graph and real timings.

Run Independent Calls Together

Run the cell. Then set parallel to false.

• In a row: 11.2 seconds.
• In parallel: 10.5 seconds. The profile and the search finish while the agent is still choosing a tool.

The gain is small here, because the longest chain is the agent loop: classify, pick a tool, read the order, reply. Parallel calls cut the wait to the longest chain, not below it.

Simple lesson: start each call as soon as its inputs exist.

Measure p95, Not the Average

Users remember the slow requests. An SLO is set on a percentile:

• p50: half of requests are faster than this
• p95: 95% are faster, the slowest 5% are not
• p99: the slowest 1%

The nearest-rank method sorts the samples and takes the value at position ceil(p ÷ 100 × n).

Simple lesson: set and check SLOs on p95, from real records.

What the User Feels

The user does not watch the total time. They watch the screen.

Run the cell.

• Parallel: a little faster.
• Streaming alone: no change, because the judge holds the reply until it has checked all of it.
• Judge after: the reply streams as it is written, and the judge checks the full text afterwards. First text at 6.0 seconds.
• Acknowledgement: a fast message such as "Checking your order…" appears at once. It helps the user wait. It is not the answer.

6.0 seconds still breaks the 4-second SLO. The rest has to come from shorter calls on the critical path: the cached prefix from Prompt Architecture and Caching, a smaller model from Routing and Escalation, and fewer agent steps.

Run npm run latency (or dotnet run -- latency). It prints the p50, p95 and SLO breaches, the critical path in a row and in parallel, and the first visible text before and after.

Simple lesson: stream, unblock the first text, and shorten the critical path.

Shorten the Tail

The SLO is on p95, so the slowest requests matter most. Three more patterns help:

• request decomposition: split one big call into small parallel calls, such as one per document
• hedged requests: if a call is not done by a deadline, send a copy to another replica and take the first answer
• speculative execution: start a likely next step early, and throw the result away if it is not needed

All three spend extra tokens to save time. Measure the extra cost.

Run the cell. Hedging at 6 seconds cuts p95 from 12.0 to 8.3 seconds for 4 extra requests in 40. Hedging at 4 seconds cuts it to 6.3 seconds, and sends 23 extra requests.

Simple lesson: hedge late, so you pay extra only for the slow tail.

Try it on a real model. Stream a real reply and run a real judge. The tab shows when the user sees text with streaming, and with a blocking judge. Switch Model to Real in the lab's top bar. It runs an open model in your browser with WebLLM, and needs WebGPU. It is not graded.

Put It in Your App

• Trace the graph. Use spans in OpenTelemetry for each step, with parent and child links. A trace viewer then shows the real critical path.
• Stream end to end. Stream from the model, through your server, to the browser. A buffering proxy can undo streaming.
• Report the first visible text. Log the time the first chunk was sent to the user, next to ttftMs and totalMs.

Simple lesson: measure what the user sees, and trace where the time goes.

Common mistakes

Check Your Understanding

Exercises

Build It in the Project

The other files of this lesson's project are optional practice. Finish them to run the lesson's script on your own code.

What to Remember

• Draw the calls as a graph, and run independent ones together.
• The critical path sets the wait, so shorten it.
• Set SLOs on p95, from real records.
• Streaming helps only if nothing blocks the first text.
• An acknowledgement helps users wait, and is not the answer.

Next: Budgets, Limits and Cost Alerts. You will stop runaway agents and catch cost spikes within an hour.