# Batch Work Course: [LLM Cost, Latency & Model Selection](https://www.devtutorials.ai/learn/llm-cost-latency-model-selection) ยท Section: Performance Source: https://www.devtutorials.ai/learn/llm-cost-latency-model-selection/performance/batch-work > Move work nobody waits for to a batch API: decide per job, write a JSONL batch file with unique ids, and merge results back in order while collecting failures, in TypeScript or C#. ## Work Nobody Is Waiting For Not every model call has a user waiting for it. Brightgrid runs five jobs a day: - **live_chat:** customers waiting in the chat window - **nightly_summaries:** a summary of every ticket, for the next morning - **tag_backlog:** tags on old tickets, needed this week - **weekly_report:** one long report for managers - **urgent_reindex:** needed within 6 hours All five use the real-time API today. Most providers also offer a **batch API**. You upload a file of requests, and get the results within a time window, often 24 hours, at a lower price. Many providers charge about half. Press **Play**, or use **Next** to go one step at a time. - **Frame 1:** every job on the real-time API, with its daily cost. - **Frame 2:** jobs that can wait go to batch. Live chat and the 6-hour job stay real-time. **Simple lesson:** if nobody waits for the answer, do not pay for speed. ### A Real-World Example The nightly summaries ran at 2 a.m. on the real-time API, one call per ticket. They hit rate limits, retried, and competed with the morning's first live chats. On the batch API the same job costs half, runs on the provider's spare capacity, and does not touch the live rate limits. **Simple lesson:** batch is cheaper, and it keeps background work away from your users. ## What You Will Build In this lesson, you will learn: 1. how to decide, per job, between real-time and batch 2. how to write a batch input file in the common JSONL format 3. how to match results back to requests, and collect the ones to retry You will finish three files in the **batch** folder: - **Plan** (`src/batch/planJobs.ts`, or `Batch/PlanJobs.cs`): real-time or batch, with the daily cost. - **Batch file** (`src/batch/batchFile.ts`, or `Batch/BatchFile.cs`): one JSON line per request. - **Merge** (`src/batch/mergeResults.ts`, or `Batch/MergeResults.cs`): results in request order, and failures. Open the **Code** tab. The harness has the five jobs, the batch price, and a sample result file from a batch run. **Simple lesson:** batch is a separate code path: plan, write the file, submit, poll, merge. ## Decide Per Job Run the cell. Live chat must stay real-time. The nightly summaries and the tag backlog together cost twice as much as live chat, and nobody waits for them. A job can use batch when **both** are true: - **no user is waiting** for the result - its **deadline** is at least the batch window, here 24 hours The urgent reindex fails the second rule. It waits for nobody, and still cannot wait 24 hours. **Simple lesson:** batch when nobody waits and the deadline allows it. ## Write the Batch File Batch APIs such as OpenAI's take a JSONL file: one JSON object per line. Each line has: - a **custom_id** that you choose, to match the result later - the **method** and **url** of the normal endpoint - the normal request **body**: model, messages, `max_tokens` Results come back with the same `custom_id`, in any order. Run the cell. The second prompt has quotes in it. A JSON library escapes them. Gluing strings together would produce a broken line. **The custom_id is your only link back.** Make it unique and stable, such as job name plus ticket id. Many batch APIs limit its length. A duplicate id means two results you cannot tell apart. **Simple lesson:** build lines with a JSON library, and make every id unique. ## Merge Results and Retry Failures A batch result file is not a clean list of answers. Look at the sample: - the results are in a **different order** from the requests - one line has an **error** object - one line has a **429** status inside the response - one request has **no line at all** Your merge code must put answers back in request order, and collect everything that failed or is missing. You send those again, in the next batch or in real time. **Some batch requests fail.** Rate limits, timeouts and server errors happen inside a batch too. The batch as a whole can say "completed" while some lines failed. Always check each line. Run `npm run batch` (or `dotnet run -- batch`). The daily cost drops by 34%, and the merge reports 2 good results and 3 to retry. **Simple lesson:** merge by id, never by position, and retry what failed. **Try it on a real model.** Tag five real tickets, then price the same tokens at the real-time and the batch rate. Switch **Model** to **Real** in the lab's top bar. It runs an open model in your browser with WebLLM, and needs WebGPU. It is not graded. ## Put It in Your App - **A job table.** Store each batch job with its id, status, input file and output file. A scheduled worker polls the status. - **Idempotent merge.** Merging the same result file twice must not create duplicate rows. - **Fallback.** If a batch is late and the deadline is close, send the remaining items in real time. - **Cost records.** Log batch calls with a `batch: true` flag and the batch price, so reports stay right. **Batch works with self-hosted models too.** A local server has no batch discount, but running background work in large groups at night uses GPUs that would otherwise sit idle. **Simple lesson:** batch needs a job table, a poller, a merge and a fallback. ## Common mistakes | Mistake | Wrong idea | Correct idea | What goes wrong | |---|---|---|---| | Batching anything that is not interactive | "Background means batch." | Check the deadline against the batch window. | Urgent work arrives late. | | Building JSON lines by string concatenation | "It is just text." | Use a JSON library. | Quotes in a prompt break the file. | | Matching results by position | "Results come back in order." | Match by `custom_id`. | Answers attach to the wrong tickets. | | Trusting "completed" | "The batch finished, so all lines worked." | Check every line's error and status. | Failed items are silently lost. | | No fallback | "Batches are never late." | Switch to real time near the deadline. | A missed morning report. | ## Check Your Understanding ## Exercises ## Build It in the Project The other files of this lesson's project are optional practice. Finish them to run the lesson's script on your own code. ## What to Remember - Batch APIs are cheaper and slower. Use them when nobody waits and the deadline allows. - A batch file is JSONL: one request per line, each with a unique `custom_id`. - Results come back in any order, and some lines fail. - Merge by id, collect failures, and retry them. - Keep a job table, a poller and a real-time fallback. Next: **Latency Engineering**. You will cut the time the user waits by running independent calls together and streaming the reply.