Orchestrator-Workers

By the Syncfusion DevTutorials team · Last updated October 20, 2026

When the work list is only found while running, an orchestrator starts workers for it. Cap width, depth and spend, record what it really started, and keep a kill switch.

Watch an Orchestrator Start Too Many Workers

The project is moving to a new API client. Every old call must change. Nobody knows how many there are until the code is searched. It could be 5. It could be 50.

An orchestrator node searches the code, finds the call sites, and starts one worker for each. Run it.

It found 18 call sites and started a worker for each. Some workers found big files and started their own workers. 33 workers called the API at once. The API allows 8. The other 25 got 429 Too Many Requests, retried at once, and hit the limit again, and again.

Nothing told the orchestrator how far it may go. In this lesson you give it limits.

TL;DR

• Orchestrator-workers is for work you can only list while running: the orchestrator finds the work and starts workers for it.
• It fails when it is unbounded: too many workers at once, workers starting workers, or no spend limit.
• Give it caps: width (how many at once), depth (may workers start workers?) and spend (tokens for the whole subtree).
• Set width at or below the limit of what the workers call, such as a rate-limited API.
• Record the realized topology: what was actually started. Add a kill switch that stops every worker at once.

A Real-World Example

Omar asks an agent to "update every test file to the new test helper". The agent finds 140 test files and starts a subagent for each. His laptop fans spin up. His API bill for the afternoon is larger than for the rest of the month. Half the subagents fail on rate limits, and some of them start subagents of their own to "split the work".

He stops everything by closing the terminal. Then he cannot tell which files were done.

The second time, the plan has limits: at most 6 workers at once, workers may not start workers, and the whole job may use at most 200,000 tokens. Each finished file is recorded in the state file. A file called GRAPH_PAUSED stops everything cleanly. The job takes a bit longer, and finishes.

What You Will Learn

• When the work list is only known at run time
• How to cap an orchestrator's width, depth and spend
• Why a rate-limited API needs a width cap
• How to record what the orchestrator really started
• How to stop every worker at once

Work Found at Run Time

Most patterns so far had a fixed shape: you knew the nodes before the run. Orchestrator-workers is different. The orchestrator discovers the work, and the number of workers depends on what it finds.

The orchestrator finds the work

const files = {
  "src/api/tickets.ts": ["fetch(", "fetch("],
  "src/api/users.ts":   ["fetch("],
  "src/lib/export.ts":  [],
  "src/App.tsx":        ["fetch(", "fetch(", "fetch("],
};

const sites = [];
for (const [file, calls] of Object.entries(files)) calls.forEach((_, i) => sites.push(file + "#" + (i + 1)));

console.log("found " + sites.length + " call sites in " + Object.keys(files).length + " files");
console.log("first three: " + sites.slice(0, 3).join(", "));

Expected output:

found 6 call sites in 4 files
first three: src/api/tickets.ts#1, src/api/tickets.ts#2, src/api/users.ts#1

This is the right pattern when the list really is unknown ahead of time and the items are independent. If you can list the work before you start, a plain fan-out (Lesson 5) is simpler.

Caps: How Wide, How Deep, How Much

An orchestrator without limits does what a busy manager with an unlimited budget does: it hires everyone at once. Three caps stop that.

In the graph file, caps go on the orchestrator node:

kill_switch: GRAPH_PAUSED
nodes:
  - { id: plan, kind: orchestrator, record_topology: true, caps: { width: 8, depth: 1, spend: 120000 }, reads: [task], writes: [call_sites] }

Width

width is how many workers run at the same time. Extra work waits in a queue. Set it at or below the limit of the thing every worker calls.

Width vs a rate limit of 8

const LIMIT = 8;
function run(workers, width) {
  const atOnce = Math.min(workers, width);
  const rejected = Math.max(0, atOnce - LIMIT);
  return atOnce + " at once, " + rejected + " rejected with 429";
}

console.log("no cap:   " + run(33, Infinity));
console.log("width 20: " + run(33, 20));
console.log("width 8:  " + run(33, 8));

Expected output:

no cap:   33 at once, 25 rejected with 429
width 20: 20 at once, 12 rejected with 429
width 8:  8 at once, 0 rejected with 429

The rejected workers are the start of the trouble. If they all retry at once, with no wait, they hit the limit again. That is a 429 storm: lots of tokens and time spent being told "no".

Depth

depth says how many levels of workers may exist. depth: 1 means: the orchestrator starts workers, and workers may not start their own workers.

Without a depth cap, a worker that finds a big file may decide to "split the work" and start sub-workers, which may start more. Each level multiplies the count.

How depth multiplies workers

const sites = 18;
const bigFiles = 3; // workers that find a big file
const perSplit = 3;

// Depth 2: each big file is split into 3 more workers.
// Depth 3: those split again, adding 2 per big file.
function workers(depth) {
  let total = sites;
  if (depth >= 2) total += bigFiles * perSplit;
  if (depth >= 3) total += bigFiles * 2;
  return total;
}

for (const d of [1, 2, 3]) console.log("depth " + d + ": " + workers(d) + " workers");

Expected output:

depth 1: 18 workers
depth 2: 27 workers
depth 3: 33 workers

Spend

spend is a token budget for the orchestrator and everything under it. When it runs out, the run stops on that bound, and the state file lists what is left. You choose how much the job may cost before you start, not after.

Record What Really Ran

The graph file says what may run. The realized topology is what did run: how many workers, at what depth, how many retries. Without it, you cannot tell a normal run from a runaway one.

A realized topology record

const record = [
  { node: "plan",   copies: 1,  depth: 0 },
  { node: "worker", copies: 18, depth: 1 },
];
const total = record.reduce((s, r) => s + r.copies, 0);
console.log(record.map((r) => r.node + " x" + r.copies + " (depth " + r.depth + ")").join(" -> "));
console.log("total nodes run: " + total + ", deepest level: " + Math.max(...record.map((r) => r.depth)));

Expected output:

plan x1 (depth 0) -> worker x18 (depth 1)
total nodes run: 19, deepest level: 1

In the lab, record_topology: true on the orchestrator saves this record with every run.

Spend for the Whole Subtree

The spend cap covers the orchestrator and every worker under it. Here is how a cap of 60,000 tokens plays out on the three versions:

Where a spend cap stops the run

const PER_SITE = 1500;
const CAP = 60000;

for (const sites of [18, 34, 52]) {
  const need = sites * PER_SITE;
  const done = need <= CAP ? sites : Math.floor(CAP / PER_SITE);
  console.log(sites + " sites: needs " + need + ", " + (done === sites ? "all done" : "stopped on the cap after " + done));
}

Expected output:

18 sites: needs 27000, all done
34 sites: needs 51000, all done
52 sites: needs 78000, stopped on the cap after 40

Check the Plan at the Join

Workers can fail quietly. A worker times out, writes nothing, and the join sees results only from the workers that finished. If the join just counts what came back, it reports "all done" for work that never happened.

The fix is a join that compares the plan with the results: the orchestrator's list of call sites against the diffs that came back. Anything missing goes back to the queue.

  - { id: join, kind: join, timeout: 15m, policy: all, check_plan: true, reads: [call_sites, diffs], writes: [summary] }

Compare the plan with the results

const plan = ["api/tickets#1", "api/tickets#2", "api/users#1", "App#1", "App#2"];
const results = ["api/tickets#1", "api/users#1", "App#1", "App#2"];

const missing = plan.filter((site) => !results.includes(site));
console.log("planned " + plan.length + ", came back " + results.length);
console.log(missing.length ? "missing: " + missing.join(", ") + " -> back to the queue" : "all done");

Expected output:

planned 5, came back 4
missing: api/tickets#2 -> back to the queue

A Kill Switch

Sometimes you need to stop everything now: the bill climbs, or the workers are breaking things. Closing a terminal is not a plan. It leaves half-finished work and no record.

A kill switch is a simple, agreed signal that every worker checks before each step, such as a file named GRAPH_PAUSED in the repo. Create the file, and every worker stops after its current step and saves its state. Delete it, and the graph can resume from the checkpoint.

Lab Exercise: Cap the Orchestrator

Open the graph file. Change it so that:

1. The orchestrator has caps: with width of 8 or less, depth: 1, and a spend budget big enough for the largest version.
2. The orchestrator has record_topology: true.
3. The join has check_plan: true and reads call_sites as well as diffs.
4. The graph has kill_switch: GRAPH_PAUSED at the top. (The linter asks for it; the check does not.)

Then press Check my graph. The 3 versions find 18, 34 and 52 call sites. In every version, two workers time out quietly. Every version must migrate all its call sites, with 0 rate-limit errors, the realized topology recorded, and no false "all done".

Common mistakes

Exercises

Each exercise ends with a value on the last line. Press Check to compare it with the expected answer.

What to Remember

• Orchestrator-workers is for work you can only list while running.
• Cap width at or below the limit of what workers call. Uncapped width causes 429 storms.
• depth: 1 stops workers from starting workers.
• A spend cap bounds the whole subtree. Size it for the biggest job.
• Record the realized topology, and keep a kill switch that stops everything cleanly.

Further Reading

• How to Implement Graph Engineering, by QA Wolf: orchestrator-workers, unbounded orchestrators, and the API-migration worked example.
• Building Effective Agents, by Anthropic: orchestrator-workers for tasks where the subtasks are not known in advance.
• Claude Code subagents: how many subagents run at once, and how to change it.

Next: Evaluator-Optimizer. A fixer goes back and forth between two broken versions, and you write the exits that stop it, then export your first graph.