# Orchestrator-Workers Course: [Graph Engineering](https://www.devtutorials.ai/learn/graph-engineering) ยท Section: Graph Foundations Source: https://www.devtutorials.ai/learn/graph-engineering/graph-foundations/orchestrator-workers > When the work list is only found while running, an orchestrator starts workers for it. Cap width, depth and spend, record what it really started, and keep a kill switch. ## Watch an Orchestrator Start Too Many Workers The project is moving to a new API client. Every old call must change. Nobody knows how many there are until the code is searched. It could be 5. It could be 50. An **orchestrator** node searches the code, finds the call sites, and starts one **worker** for each. Run it. It found 18 call sites and started a worker for each. Some workers found big files and started **their own** workers. 33 workers called the API at once. The API allows 8. The other 25 got `429 Too Many Requests`, retried at once, and hit the limit again, and again. Nothing told the orchestrator how far it may go. In this lesson you give it limits. ## TL;DR - **Orchestrator-workers** is for work you can only list **while running**: the orchestrator finds the work and starts workers for it. - It fails when it is **unbounded**: too many workers at once, workers starting workers, or no spend limit. - Give it **caps**: `width` (how many at once), `depth` (may workers start workers?) and `spend` (tokens for the whole subtree). - Set `width` at or below the limit of what the workers call, such as a rate-limited API. - **Record the realized topology**: what was actually started. Add a **kill switch** that stops every worker at once. ## A Real-World Example Omar asks an agent to "update every test file to the new test helper". The agent finds 140 test files and starts a subagent for each. His laptop fans spin up. His API bill for the afternoon is larger than for the rest of the month. Half the subagents fail on rate limits, and some of them start subagents of their own to "split the work". He stops everything by closing the terminal. Then he cannot tell which files were done. The second time, the plan has limits: at most 6 workers at once, workers may not start workers, and the whole job may use at most 200,000 tokens. Each finished file is recorded in the state file. A file called `GRAPH_PAUSED` stops everything cleanly. The job takes a bit longer, and finishes. ## What You Will Learn - When the work list is only known at run time - How to cap an orchestrator's width, depth and spend - Why a rate-limited API needs a width cap - How to record what the orchestrator really started - How to stop every worker at once ## Work Found at Run Time Most patterns so far had a fixed shape: you knew the nodes before the run. Orchestrator-workers is different. The orchestrator **discovers** the work, and the number of workers depends on what it finds. ```js runnable title="The orchestrator finds the work" const files = { "src/api/tickets.ts": ["fetch(", "fetch("], "src/api/users.ts": ["fetch("], "src/lib/export.ts": [], "src/App.tsx": ["fetch(", "fetch(", "fetch("], }; const sites = []; for (const [file, calls] of Object.entries(files)) calls.forEach((_, i) => sites.push(file + "#" + (i + 1))); console.log("found " + sites.length + " call sites in " + Object.keys(files).length + " files"); console.log("first three: " + sites.slice(0, 3).join(", ")); ``` Expected output: ```text found 6 call sites in 4 files first three: src/api/tickets.ts#1, src/api/tickets.ts#2, src/api/users.ts#1 ``` This is the right pattern when the list really is unknown ahead of time and the items are independent. If you can list the work before you start, a plain fan-out (Lesson 5) is simpler. ## Caps: How Wide, How Deep, How Much An orchestrator without limits does what a busy manager with an unlimited budget does: it hires everyone at once. Three caps stop that. In the graph file, caps go on the orchestrator node: ```yaml kill_switch: GRAPH_PAUSED nodes: - { id: plan, kind: orchestrator, record_topology: true, caps: { width: 8, depth: 1, spend: 120000 }, reads: [task], writes: [call_sites] } ``` ### Width `width` is how many workers run **at the same time**. Extra work waits in a queue. Set it at or below the limit of the thing every worker calls. ```js runnable title="Width vs a rate limit of 8" const LIMIT = 8; function run(workers, width) { const atOnce = Math.min(workers, width); const rejected = Math.max(0, atOnce - LIMIT); return atOnce + " at once, " + rejected + " rejected with 429"; } console.log("no cap: " + run(33, Infinity)); console.log("width 20: " + run(33, 20)); console.log("width 8: " + run(33, 8)); ``` Expected output: ```text no cap: 33 at once, 25 rejected with 429 width 20: 20 at once, 12 rejected with 429 width 8: 8 at once, 0 rejected with 429 ``` The rejected workers are the start of the trouble. If they all retry at once, with no wait, they hit the limit again. That is a **429 storm**: lots of tokens and time spent being told "no". ### Depth `depth` says how many levels of workers may exist. `depth: 1` means: the orchestrator starts workers, and workers may **not** start their own workers. Without a depth cap, a worker that finds a big file may decide to "split the work" and start sub-workers, which may start more. Each level multiplies the count. ```js runnable title="How depth multiplies workers" const sites = 18; const bigFiles = 3; // workers that find a big file const perSplit = 3; // Depth 2: each big file is split into 3 more workers. // Depth 3: those split again, adding 2 per big file. function workers(depth) { let total = sites; if (depth >= 2) total += bigFiles * perSplit; if (depth >= 3) total += bigFiles * 2; return total; } for (const d of [1, 2, 3]) console.log("depth " + d + ": " + workers(d) + " workers"); ``` Expected output: ```text depth 1: 18 workers depth 2: 27 workers depth 3: 33 workers ``` ### Spend `spend` is a token budget for the orchestrator **and everything under it**. When it runs out, the run stops on that bound, and the state file lists what is left. You choose how much the job may cost before you start, not after. A spend cap that is too low is also a bug. It stops every run before the work is done. Size it from the biggest job you expect, with some room, and treat "stopped on the spend cap" as a signal to look, not as a normal finish. ## Record What Really Ran The graph file says what **may** run. The **realized topology** is what **did** run: how many workers, at what depth, how many retries. Without it, you cannot tell a normal run from a runaway one. ```js runnable title="A realized topology record" const record = [ { node: "plan", copies: 1, depth: 0 }, { node: "worker", copies: 18, depth: 1 }, ]; const total = record.reduce((s, r) => s + r.copies, 0); console.log(record.map((r) => r.node + " x" + r.copies + " (depth " + r.depth + ")").join(" -> ")); console.log("total nodes run: " + total + ", deepest level: " + Math.max(...record.map((r) => r.depth))); ``` Expected output: ```text plan x1 (depth 0) -> worker x18 (depth 1) total nodes run: 19, deepest level: 1 ``` In the lab, `record_topology: true` on the orchestrator saves this record with every run. ### Spend for the Whole Subtree The spend cap covers the orchestrator **and** every worker under it. Here is how a cap of 60,000 tokens plays out on the three versions: ```js runnable title="Where a spend cap stops the run" const PER_SITE = 1500; const CAP = 60000; for (const sites of [18, 34, 52]) { const need = sites * PER_SITE; const done = need <= CAP ? sites : Math.floor(CAP / PER_SITE); console.log(sites + " sites: needs " + need + ", " + (done === sites ? "all done" : "stopped on the cap after " + done)); } ``` Expected output: ```text 18 sites: needs 27000, all done 34 sites: needs 51000, all done 52 sites: needs 78000, stopped on the cap after 40 ``` ## Check the Plan at the Join Workers can fail quietly. A worker times out, writes nothing, and the join sees results only from the workers that finished. If the join just counts what came back, it reports "all done" for work that never happened. The fix is a join that compares the **plan** with the **results**: the orchestrator's list of call sites against the diffs that came back. Anything missing goes back to the queue. ```yaml - { id: join, kind: join, timeout: 15m, policy: all, check_plan: true, reads: [call_sites, diffs], writes: [summary] } ``` ```js runnable title="Compare the plan with the results" const plan = ["api/tickets#1", "api/tickets#2", "api/users#1", "App#1", "App#2"]; const results = ["api/tickets#1", "api/users#1", "App#1", "App#2"]; const missing = plan.filter((site) => !results.includes(site)); console.log("planned " + plan.length + ", came back " + results.length); console.log(missing.length ? "missing: " + missing.join(", ") + " -> back to the queue" : "all done"); ``` Expected output: ```text planned 5, came back 4 missing: api/tickets#2 -> back to the queue ``` ## A Kill Switch Sometimes you need to stop everything **now**: the bill climbs, or the workers are breaking things. Closing a terminal is not a plan. It leaves half-finished work and no record. A **kill switch** is a simple, agreed signal that every worker checks before each step, such as a file named `GRAPH_PAUSED` in the repo. Create the file, and every worker stops after its current step and saves its state. Delete it, and the graph can resume from the checkpoint. In the real tools, the orchestrator is usually the main agent following `graph.md`, and workers are subagents. Claude Code runs up to 20 subagents at once by default; Codex has a setting for how many agent threads may run at once. Neither knows your API's rate limit. Write the width cap in `graph.md` and in each worker's instructions. ## Lab Exercise: Cap the Orchestrator Open the graph file. Change it so that: 1. The orchestrator has `caps:` with `width` of 8 or less, `depth: 1`, and a `spend` budget big enough for the largest version. 2. The orchestrator has `record_topology: true`. 3. The join has `check_plan: true` and reads `call_sites` as well as `diffs`. 4. The graph has `kill_switch: GRAPH_PAUSED` at the top. (The linter asks for it; the check does not.) Then press **Check my graph**. The **3 versions** find 18, 34 and 52 call sites. In every version, two workers time out quietly. Every version must migrate all its call sites, with 0 rate-limit errors, the realized topology recorded, and no false "all done". For the spend cap: each call site costs about 1,500 tokens, and the biggest version has 52. Leave some room above that. ## Common mistakes | Mistake | Wrong idea | Correct idea | What goes wrong | |---|---|---|---| | No width cap | "More workers finish sooner." | Width at or below the API's limit. | A 429 storm: retries fail again and again. | | Width above the limit | "The API limit is just a guideline." | The limit is a hard ceiling. | Every extra worker is rejected. | | No depth cap | "Workers know when to split." | `depth: 1` by default. | Workers multiply level by level. | | Spend cap too low | "A tiny budget is safest." | Size it for the biggest job, with room. | Every big run stops halfway. | | No record of what ran | "The graph file says what runs." | The realized topology says what did run. | A runaway looks like a normal run. | ## Exercises Each exercise ends with a value on the last line. Press **Check** to compare it with the expected answer. ## What to Remember - Orchestrator-workers is for work you can only list while running. - Cap width at or below the limit of what workers call. Uncapped width causes 429 storms. - `depth: 1` stops workers from starting workers. - A spend cap bounds the whole subtree. Size it for the biggest job. - Record the realized topology, and keep a kill switch that stops everything cleanly. ## Further Reading - [How to Implement Graph Engineering, by QA Wolf](https://www.qawolf.com/blog/how-to-implement-graph-engineering-5-composable-patterns): orchestrator-workers, unbounded orchestrators, and the API-migration worked example. - [Building Effective Agents, by Anthropic](https://www.anthropic.com/engineering/building-effective-agents): orchestrator-workers for tasks where the subtasks are not known in advance. - [Claude Code subagents](https://code.claude.com/docs/en/subagents): how many subagents run at once, and how to change it. Next: **Evaluator-Optimizer**. A fixer goes back and forth between two broken versions, and you write the exits that stop it, then export your first graph.