Ship It: Flags, Canaries and Rollbacks

By Syncfusion®, Inc. · Last updated

Route between the baseline and the fine-tune with a flag, run shadow and canary steps with an automatic rule, fall back on bad answers, and block unapproved releases in CI.

Ship It: Flags, Canaries and Rollbacks

The fine-tune passed the eval. Someone signed the manifest. Now it has to reach customers without breaking anything.

Switching every request to the new model at once is a bet. If it fails on traffic the eval never saw, everyone sees it at the same moment.

Teams that ship models safely do it in steps: 1, 5, 25, 50, 100 percent, with a rule that decides each step and a fallback that is always one switch away.

Simple lesson: ship a model like a risky code change: behind a flag, a little at a time, with the old version ready.

A Real-World Example

Think about a restaurant adding a new dish.

The chef first cooks it in the kitchen for the staff, without serving it. Then a few tables get it as a special. If they send it back, it comes off the menu that night. Only when it keeps working does it go on the full menu.

And the old dish stays in the kitchen until the new one has proved itself.

Simple lesson: try it where it cannot hurt, then where it can only hurt a little.

What You Will Learn

In this lesson, you will learn:

  1. The four routing modes: off, shadow, canary and full.
  2. How to pick canary customers with a stable hash, so nobody flips between models.
  3. How to fall back to the baseline when a fine-tune answer is broken.
  4. How a canary rule decides promote, hold or roll back.
  5. How a model manifest and a CI release check stop an unapproved model.

Before you start

  • Feature flag = a setting that changes what the code does without a deploy. Here it is ROUTE_MODE and CANARY_PERCENT.
  • Baseline = the prompt-only model you had before the fine-tune. It stays deployed as the fallback.
  • p95 latency = the time 95% of requests finish within. One slow request in twenty is slower than this.
  • The code here is a short version of the build-along repo's router, rollout and manifest files, in TypeScript and C#.

TL;DR

  • off: the baseline answers. shadow: the baseline answers and the fine-tune runs on the side, only logged. canary: some customers get the fine-tune. full: everyone does.
  • Pick canary customers with a hash of their ID, never with a random number.
  • Any fine-tune answer that fails the shape check, times out or errors falls back to the baseline.
  • Step up only when a rule says so. Roll back when the fine-tune fails more than 2 points above the baseline or is more than 1.5 times slower.
  • CI blocks a release without eval numbers and a named sign-off.

Pick Customers with a Stable Hash

A customer should see the same model on every request. A random number would switch them back and forth, and their support ticket would change format halfway through a conversation.

Which customers get the fine-tune

// Which customers get the fine-tune? A stable hash, never a random number.
type Mode = "off" | "shadow" | "canary" | "full";

function bucket(customerId: string): number {   // FNV-1a, 0-99
  let h = 0x811c9dc5;
  for (let i = 0; i < customerId.length; i++) { h ^= customerId.charCodeAt(i); h = Math.imul(h, 0x01000193) >>> 0; }
  return h % 100;
}
function useTuned(mode: Mode, percent: number, customerId: string): boolean {
  if (mode === "full") return true;
  if (mode === "canary") return bucket(customerId) < percent;
  return false;   // off, and shadow: the baseline answers
}

console.log(`C-0042 is in bucket ${bucket("C-0042")}, every time`);
const customers = Array.from({ length: 10000 }, (_, i) => `C-${10000 + i}`);
for (const percent of [1, 5, 25]) {
  const n = customers.filter((c) => useTuned("canary", percent, c)).length;
  console.log(`canary ${String(percent).padStart(2)}%: ${n} of ${customers.length} customers`);
}
const in5 = customers.filter((c) => useTuned("canary", 5, c));
console.log(`all ${in5.length} customers from 5% are still in at 25%: ${in5.every((c) => useTuned("canary", 25, c))}`);

Which customers get the fine-tune

// Which customers get the fine-tune? A stable hash, never a random number.
int Bucket(string customerId)   // FNV-1a, 0-99
{
    uint h = 0x811c9dc5;
    foreach (var ch in customerId) { h ^= ch; h = unchecked(h * 0x01000193); }
    return (int)(h % 100);
}
bool UseTuned(string mode, int percent, string customerId) => mode switch
{
    "full" => true,
    "canary" => Bucket(customerId) < percent,
    _ => false,   // off, and shadow: the baseline answers
};

Console.WriteLine($"C-0042 is in bucket {Bucket("C-0042")}, every time");
var customers = Enumerable.Range(0, 10000).Select(i => $"C-{10000 + i}").ToList();
foreach (var percent in new[] { 1, 5, 25 })
{
    var n = customers.Count(c => UseTuned("canary", percent, c));
    Console.WriteLine($"canary {percent,2}%: {n} of {customers.Count} customers");
}
var in5 = customers.Where(c => UseTuned("canary", 5, c)).ToList();
Console.WriteLine($"all {in5.Count} customers from 5% are still in at 25%: {in5.All(c => UseTuned("canary", 25, c)).ToString().ToLowerInvariant()}");

Expected output:

C-0042 is in bucket 15, every time
canary  1%: 92 of 10000 customers
canary  5%: 503 of 10000 customers
canary 25%: 2509 of 10000 customers
all 503 customers from 5% are still in at 25%: true

Raising the canary from 5% to 25% only adds customers. Nobody who had the fine-tune loses it. The repo's TypeScript and C# services use the same hash, so they agree on every customer.

Fall Back on Every Bad Answer

Users should never see a broken ticket. The router checks the fine-tune's answer with the same shape check as the eval, and asks the baseline when it fails.

Fall back when the fine-tune's answer is broken

// Route, check the shape, fall back. Every request writes one log line.
type Line = { requestId: string; path: string; ok: boolean; why: string };

function problem(raw: string): string {   // a short version of the repo's shape check
  const t = raw.trim();
  if (!t.startsWith("{")) return "not JSON (prose)";
  const end = t.indexOf("}") + 1;
  try { JSON.parse(t.slice(0, end)); } catch { return "truncated or invalid JSON"; }
  return t.slice(end).trim() ? "extra text after the JSON" : "";
}

const tunedAnswers: Record<string, string> = {
  r1: '{"intent":"refund","order":"2518"}',
  r2: '{"intent":"where","order":"2604"}\nAnything else?',
  r3: '{"intent":"cancel","ord',
};
const baselineAnswer = '{"intent":"refund","order":"2518"}';
const log: Line[] = [];

function handle(requestId: string): string {
  const raw = tunedAnswers[requestId], why = problem(raw);
  log.push({ requestId, path: "tuned", ok: !why, why });
  if (!why) return "tuned";
  log.push({ requestId, path: "fallback", ok: problem(baselineAnswer) === "", why: "" });
  return "fallback";
}

for (const id of ["r1", "r2", "r3"]) console.log(`${id}: answered by ${handle(id)}`);
console.log("--- log");
for (const l of log) console.log(JSON.stringify(l));

Fall back when the fine-tune's answer is broken

using System.Text.Json;

// Route, check the shape, fall back. Every request writes one log line.
string Problem(string raw)   // a short version of the repo's shape check
{
    var t = raw.Trim();
    if (!t.StartsWith('{')) return "not JSON (prose)";
    var end = t.IndexOf('}') + 1;
    try { JsonDocument.Parse(t[..end]); } catch (JsonException) { return "truncated or invalid JSON"; }
    return t[end..].Trim().Length > 0 ? "extra text after the JSON" : "";
}

var tunedAnswers = new Dictionary<string, string>
{
    ["r1"] = """{"intent":"refund","order":"2518"}""",
    ["r2"] = "{\"intent\":\"where\",\"order\":\"2604\"}\nAnything else?",
    ["r3"] = """{"intent":"cancel","ord""",
};
var baselineAnswer = """{"intent":"refund","order":"2518"}""";
var log = new List<Line>();

string Handle(string requestId)
{
    var raw = tunedAnswers[requestId];
    var why = Problem(raw);
    log.Add(new(requestId, "tuned", why == "", why));
    if (why == "") return "tuned";
    log.Add(new(requestId, "fallback", Problem(baselineAnswer) == "", ""));
    return "fallback";
}

foreach (var id in new[] { "r1", "r2", "r3" }) Console.WriteLine($"{id}: answered by {Handle(id)}");
Console.WriteLine("--- log");
var web = new JsonSerializerOptions(JsonSerializerDefaults.Web);
foreach (var l in log) Console.WriteLine(JsonSerializer.Serialize(l, web));

record Line(string RequestId, string Path, bool Ok, string Why);

Expected output:

r1: answered by tuned
r2: answered by fallback
r3: answered by fallback
--- log
{"requestId":"r1","path":"tuned","ok":true,"why":""}
{"requestId":"r2","path":"tuned","ok":false,"why":"extra text after the JSON"}
{"requestId":"r2","path":"fallback","ok":true,"why":""}
{"requestId":"r3","path":"tuned","ok":false,"why":"truncated or invalid JSON"}
{"requestId":"r3","path":"fallback","ok":true,"why":""}

Every request writes one log line: which path answered, whether it was well formed, and why not. These lines are what the canary rule counts, and what you read when something goes wrong.

Shadow mode first, always.

In shadow mode the fine-tune sees real traffic, but users only see the baseline. A week of shadow logs tells you its real failure rate and latency before a single customer depends on it.

Decide Each Step with a Rule

The rule compares the fine-tune with the baseline on the same window of traffic. It needs at least 500 requests before it promotes.

Decide the next canary step

// One window of live numbers -> promote, hold or roll back.
type Window = { name: string; requests: number; tunedFails: number; baselineRequests: number; baselineFails: number; tunedP95Ms: number; baselineP95Ms: number };
const policy = { minRequests: 500, maxExtraFailRate: 0.02, maxLatencyRatio: 1.5, steps: [1, 5, 25, 50, 100] };

function decide(w: Window): string {
  const tunedRate = w.tunedFails / w.requests, baseRate = w.baselineFails / w.baselineRequests;
  if (tunedRate > baseRate + policy.maxExtraFailRate) return `ROLLBACK: fail rate ${(tunedRate * 100).toFixed(1)}% vs baseline ${(baseRate * 100).toFixed(1)}%`;
  if (w.tunedP95Ms > w.baselineP95Ms * policy.maxLatencyRatio) return `ROLLBACK: p95 ${w.tunedP95Ms} ms vs baseline ${w.baselineP95Ms} ms`;
  if (w.requests < policy.minRequests) return `HOLD: only ${w.requests} of ${policy.minRequests} requests seen`;
  return "PROMOTE";
}

const current = 5;
const windows: Window[] = [
  { name: "healthy", requests: 812, tunedFails: 9, baselineRequests: 15400, baselineFails: 171, tunedP95Ms: 640, baselineP95Ms: 590 },
  { name: "too early", requests: 120, tunedFails: 1, baselineRequests: 15400, baselineFails: 171, tunedP95Ms: 640, baselineP95Ms: 590 },
  { name: "failing", requests: 790, tunedFails: 61, baselineRequests: 15100, baselineFails: 166, tunedP95Ms: 655, baselineP95Ms: 585 },
  { name: "slow", requests: 812, tunedFails: 9, baselineRequests: 15400, baselineFails: 171, tunedP95Ms: 1200, baselineP95Ms: 590 },
];
for (const w of windows) {
  const d = decide(w);
  const next = policy.steps.find((s) => s > current) ?? current;
  console.log(`${w.name.padEnd(10)} ${d === "PROMOTE" ? `PROMOTE: ${current}% -> ${next}%` : d}`);
}

Decide the next canary step

using System.Globalization;

// One window of live numbers -> promote, hold or roll back.
var policy = new { MinRequests = 500, MaxExtraFailRate = 0.02, MaxLatencyRatio = 1.5, Steps = new[] { 1, 5, 25, 50, 100 } };

string Decide(Window w)
{
    double tunedRate = (double)w.TunedFails / w.Requests, baseRate = (double)w.BaselineFails / w.BaselineRequests;
    if (tunedRate > baseRate + policy.MaxExtraFailRate)
        return string.Format(CultureInfo.InvariantCulture, "ROLLBACK: fail rate {0:F1}% vs baseline {1:F1}%", tunedRate * 100, baseRate * 100);
    if (w.TunedP95Ms > w.BaselineP95Ms * policy.MaxLatencyRatio) return $"ROLLBACK: p95 {w.TunedP95Ms} ms vs baseline {w.BaselineP95Ms} ms";
    if (w.Requests < policy.MinRequests) return $"HOLD: only {w.Requests} of {policy.MinRequests} requests seen";
    return "PROMOTE";
}

var current = 5;
var windows = new List<Window>
{
    new("healthy", 812, 9, 15400, 171, 640, 590),
    new("too early", 120, 1, 15400, 171, 640, 590),
    new("failing", 790, 61, 15100, 166, 655, 585),
    new("slow", 812, 9, 15400, 171, 1200, 590),
};
foreach (var w in windows)
{
    var d = Decide(w);
    var next = policy.Steps.Where(s => s > current).DefaultIfEmpty(current).First();
    Console.WriteLine($"{w.Name,-10} {(d == "PROMOTE" ? $"PROMOTE: {current}% -> {next}%" : d)}");
}

record Window(string Name, int Requests, int TunedFails, int BaselineRequests, int BaselineFails, int TunedP95Ms, int BaselineP95Ms);

Expected output:

healthy    PROMOTE: 5% -> 25%
too early  HOLD: only 120 of 500 requests seen
failing    ROLLBACK: fail rate 7.7% vs baseline 1.1%
slow       ROLLBACK: p95 1200 ms vs baseline 590 ms

The failing window has a 7.7% failure rate against the baseline's 1.1%, so it rolls back. The healthy one promotes 5% to 25%. In the repo, rollout:check prints this and exits 2 on a rollback, so a scheduled job can act on it.

Why compare rates, not counts

  • The baseline handles almost all the traffic during a canary, so it has many more failures in total.
  • Divide failures by requests for each side, then compare the two rates.
  • A fixed limit, such as 2 points above the baseline, keeps random noise from triggering rollbacks.

Version Everything, Check Before Release

versions/model-manifest.json records what made the model: the model ID, the base model, hashes of the training data, the config and the system prompt, the eval numbers, and who signed it off.

Which changes need a new eval

// Compare the released manifest with the new one. Some changes need a new eval.
type Manifest = { version: string; baseModel: string; datasetSha256: string; configSha256: string; systemPromptSha256: string };

function needsReeval(prev: Manifest, next: Manifest): string[] {
  const out: string[] = [];
  if (prev.baseModel !== next.baseModel) out.push("base model changed");
  if (prev.datasetSha256 !== next.datasetSha256) out.push("training data changed");
  if (prev.configSha256 !== next.configSha256) out.push("training config changed");
  if (prev.systemPromptSha256 !== next.systemPromptSha256) out.push("system prompt changed");
  return out;
}

const v1: Manifest = { version: "tickets-v1", baseModel: "gpt-4o-mini-2024-07-18", datasetSha256: "3b1f...", configSha256: "a07c...", systemPromptSha256: "5d2e..." };
const changes: [string, Partial<Manifest>][] = [
  ["new tickets added", { datasetSha256: "8e44..." }],
  ["prompt reworded", { systemPromptSha256: "c9a1..." }],
  ["provider retires the base", { baseModel: "gpt-4.1-mini-2025-04-14", configSha256: "f3b2..." }],
  ["README edited", {}],
];
for (const [what, diff] of changes) {
  const reasons = needsReeval(v1, { ...v1, ...diff, version: "tickets-v2" });
  console.log(`${what.padEnd(26)} ${reasons.length ? "RE-RUN THE EVAL: " + reasons.join(", ") : "no new eval needed"}`);
}

Which changes need a new eval

// Compare the released manifest with the new one. Some changes need a new eval.
List<string> NeedsReeval(Manifest prev, Manifest next)
{
    var o = new List<string>();
    if (prev.BaseModel != next.BaseModel) o.Add("base model changed");
    if (prev.DatasetSha256 != next.DatasetSha256) o.Add("training data changed");
    if (prev.ConfigSha256 != next.ConfigSha256) o.Add("training config changed");
    if (prev.SystemPromptSha256 != next.SystemPromptSha256) o.Add("system prompt changed");
    return o;
}

var v1 = new Manifest("tickets-v1", "gpt-4o-mini-2024-07-18", "3b1f...", "a07c...", "5d2e...");
var changes = new List<(string What, Manifest Next)>
{
    ("new tickets added", v1 with { DatasetSha256 = "8e44..." }),
    ("prompt reworded", v1 with { SystemPromptSha256 = "c9a1..." }),
    ("provider retires the base", v1 with { BaseModel = "gpt-4.1-mini-2025-04-14", ConfigSha256 = "f3b2..." }),
    ("README edited", v1),
};
foreach (var (what, next) in changes)
{
    var reasons = NeedsReeval(v1, next with { Version = "tickets-v2" });
    Console.WriteLine($"{what,-26} {(reasons.Count > 0 ? "RE-RUN THE EVAL: " + string.Join(", ", reasons) : "no new eval needed")}");
}

record Manifest(string Version, string BaseModel, string DatasetSha256, string ConfigSha256, string SystemPromptSha256);

Expected output:

new tickets added          RE-RUN THE EVAL: training data changed
prompt reworded            RE-RUN THE EVAL: system prompt changed
provider retires the base  RE-RUN THE EVAL: base model changed, training config changed
README edited              no new eval needed

In CI, the release job runs the release check on every v* tag. It fails when the eval is below the bar, the improvement interval includes zero, the safety slice failed, or approvedBy is empty.

The Workflow in CI

# .github/workflows/ci.yml (short version)
jobs:
  pipeline:   # Python: kit tests, prepare data, score.py eval gate
  app-ts:     # npm ci, typecheck, tests, data check on train.jsonl
  app-csharp: # dotnet build, tests
  release:    # on v* tags: release check on the model manifest
    needs: [pipeline, app-ts, app-csharp]

Every push runs the first three. A tag runs all four. The full file is in the repo.

What a lead owns at release time

  • The sign-off. A named person, written in the manifest. Not "the team".
  • The rollout policy: the steps, the minimum traffic and the rollback limits, in config and reviewed like code.
  • Who gets paged when the canary rolls back, and how fast someone looks.
  • How long the baseline stays deployed after full rollout. Weeks, not hours.

Do's and Don'ts

  • Do start every new model in shadow mode.
  • Do pick canary customers with a stable hash.
  • Do fall back to the baseline on any bad answer.
  • Do let a script decide promote, hold or roll back.
  • Do block releases without a named sign-off.
  • Don't switch all traffic in one step.
  • Don't remove the baseline right after full rollout.
  • Don't let a deadline replace the release check.

Common mistakes

MistakeWrong ideaCorrect ideaWhat goes wrong
Random canary"Math.random() under 5% is 5%."Hash the customer ID.Customers flip between models mid-conversation.
No fallback"The eval passed, it will be fine."Check every answer, fall back on failure.Broken tickets reach users.
Counting failures"The baseline failed 171 times, the canary 9."Compare rates.A bad canary looks better than the baseline.
Promoting early"Fifty requests look fine."Wait for enough traffic.One lucky window promotes a bad model.
Deleting the baseline"We are at 100%, clean up."Keep it as the fallback.No quick way back when drift hits.

Check Your Understanding

Exercises

What to Remember

  • Route with a flag: off, shadow, canary, full.
  • Pick canary customers with a stable hash.
  • Fall back to the baseline on any bad answer, and log every request.
  • A rule decides promote, hold or roll back, from rates over enough traffic.
  • A manifest and a release check stop an unapproved model.

Next: Capstone: Decision Record and Dataset Kit, where you bring every step together.