Routing and Escalation

By Syncfusion®, Inc. · Last updated November 2, 2026

Send requests to a small model first, validate the reply in code, and escalate on failed checks, low confidence or critical slices. Prove the router with a report on cost, quality and silent errors, in TypeScript or C#.

What you will learn

• Validate a structured model reply in code
• Route requests: accept, or escalate on a failed check, low confidence or a critical slice
• Report a router's cost, accuracy, escalation rate and silent errors

Small Model First

In Choosing a Model, each task got one model. But inside one task, most requests are easy, and a few are hard.

A router sends every request to a small model first. Code then checks the reply. When a check fails, the request escalates to the big model.

The risk is the request that looks easy, gets a confident wrong answer, and passes every check. That is a silent error, and the router's job is to have none on the slices that matter.

Press Play, or use Next to go one step at a time.

• Frame 1: an easy request stays on the small model.
• Frame 2: the validator catches a wrong order id, and the big model answers.
• Frame 3: a refund is valid and confident, and still wrong. Refunds always escalate.

Simple lesson: cheap first, checked by code, and escalated when the check or the risk says so.

A Real-World Example

A team routed by confidence only: "accept when the small model is over 80% sure". Costs halved. Then a customer got a $250 refund on a $25 order. The small model was 93% sure.

A validator would have caught it: the amount is above the order total.

Simple lesson: confidence is not correctness, so validate what you can in code.

What You Will Build

In this lesson, you will learn:

1. how to validate a structured reply in code
2. how to route: accept, or escalate on a failed check, low confidence or a critical slice
3. how to report a router's cost, quality and silent errors

You will finish three files in the routing folder:

• Validate (src/routing/validate.ts, or Routing/Validate.cs): the checks.
• Router (src/routing/router.ts, or Routing/Router.cs): accept or escalate.
• Report (src/routing/report.ts, or Routing/RouterReport.cs): cost, accuracy and silent loss.

Open the Code tab. The harness has 40 recorded support requests. Each has the small model's JSON reply, its confidence, and eval labels for both models.

Simple lesson: a router is tested on recorded traffic with labels, before it sees users.

Validate the Reply

The small model returns JSON like {"intent":"order_status","orderId":"A-1001"}. Code can check a lot of it:

• invalid_json: it does not parse
• missing_field: intent or orderId is missing
• wrong_order: the order id is not the one in the request
• refund_over_total: the refund is more than the order cost

Run the cell. Each check turns a possible silent error into an escalation.

Simple lesson: every fact you can check in code is a check the model cannot fail silently.

Route: Accept or Escalate

Escalate when any of these is true:

• the validator found a problem
• the small model's confidence is below a threshold
• the request is in a critical slice, such as refunds, where a wrong answer costs real money

Otherwise, accept the small model's reply.

A cascade pays for the small call and the big call when it escalates. Routing only saves money when most requests stay small.

Run the cell. Change the thresholds.

• 0.50: cheapest, but silent errors appear.
• 0.80: no silent errors, and still about half the cost.
• 0.93: most requests escalate, and the cascade costs almost as much as the big model alone.

Simple lesson: escalate by check, by confidence and by business risk.

Report the Router

The report is what you show before you ship the router:

• cost against the big model for everything
• accuracy against the big model's accuracy
• escalation rate
• silent errors: accepted replies that are wrong
• critical regressions: critical-slice requests the router got wrong but the big model got right

Run npm run router (or dotnet run -- router). 9 of 40 requests escalate. Cost drops by 52%, accuracy matches the big model, and there are no silent errors and no refund regressions.

Simple lesson: ship a router only with a report that shows cost, quality and zero critical regressions.

Fallbacks When a Provider Fails

Routing also handles failure. Providers rate-limit (HTTP 429) and sometimes go down (HTTP 5xx). A router with a fallback chain keeps answering:

• on 429, wait and retry once, then fall back
• on 5xx or a timeout, fall back at once
• the fallback can be another provider with the same open model, or a bigger model

Some teams also add a difficulty classifier before the small model: a cheap check that sends obviously hard requests straight to the big model, and saves the wasted small call.

Run the cell. Every request is answered, for 20% more cost than the happy path. Set useFallback to false and count the failures.

Simple lesson: a router without a fallback turns a provider outage into your outage.

Try it on a real model. Run the cascade for real. The 0.5B model answers in JSON, a validator checks it, and a failed check escalates to the 1.5B model. Switch Model to Real in the lab's top bar. It runs an open model in your browser with WebLLM, and needs WebGPU. It is not graded.

Put It in Your App

• Log the decision. Add the route, the reason and both model ids to the cost record.
• Sample the accepted replies. Send a small share of accepted replies to the eval or to a human, to catch new silent errors.
• Re-run the report on new recorded traffic when the small model, the prompt or the threshold changes.

Simple lesson: a router is a policy, logged and tested like any other.

Common mistakes

Check Your Understanding

Exercises

Build It in the Project

The other files of this lesson's project are optional practice. Finish them to run the lesson's script on your own code.

What to Remember

• Send requests to a small model first, and validate the reply in code.
• Escalate on a failed check, on low confidence, or for a critical slice.
• A cascade pays both calls when it escalates.
• Report cost, accuracy, escalation rate, silent errors and critical regressions.
• Zero regressions on critical slices is the release rule.

Next: Batch Work. You will move the work nobody is waiting for to a cheaper, slower API.