Batch Work
By Syncfusion®, Inc. · Last updated
Move work nobody waits for to a batch API: decide per job, write a JSONL batch file with unique ids, and merge results back in order while collecting failures, in TypeScript or C#.
What you will learn
- Decide per job between the real-time API and a batch API
- Write a JSONL batch input file with unique custom ids
- Merge batch results back in request order and collect failures to retry
Work Nobody Is Waiting For
Not every model call has a user waiting for it. Brightgrid runs five jobs a day:
- live_chat: customers waiting in the chat window
- nightly_summaries: a summary of every ticket, for the next morning
- tag_backlog: tags on old tickets, needed this week
- weekly_report: one long report for managers
- urgent_reindex: needed within 6 hours
All five use the real-time API today.
Most providers also offer a batch API. You upload a file of requests, and get the results within a time window, often 24 hours, at a lower price. Many providers charge about half.
Press Play, or use Next to go one step at a time.
- Frame 1: every job on the real-time API, with its daily cost.
- Frame 2: jobs that can wait go to batch. Live chat and the 6-hour job stay real-time.
Simple lesson: if nobody waits for the answer, do not pay for speed.
A Real-World Example
The nightly summaries ran at 2 a.m. on the real-time API, one call per ticket. They hit rate limits, retried, and competed with the morning's first live chats.
On the batch API the same job costs half, runs on the provider's spare capacity, and does not touch the live rate limits.
Simple lesson: batch is cheaper, and it keeps background work away from your users.
What You Will Build
In this lesson, you will learn:
- how to decide, per job, between real-time and batch
- how to write a batch input file in the common JSONL format
- how to match results back to requests, and collect the ones to retry
You will finish three files in the batch folder:
- Plan (
src/batch/planJobs.ts, orBatch/PlanJobs.cs): real-time or batch, with the daily cost. - Batch file (
src/batch/batchFile.ts, orBatch/BatchFile.cs): one JSON line per request. - Merge (
src/batch/mergeResults.ts, orBatch/MergeResults.cs): results in request order, and failures.
Open the Code tab. The harness has the five jobs, the batch price, and a sample result file from a batch run.
Simple lesson: batch is a separate code path: plan, write the file, submit, poll, merge.
Decide Per Job
Run the cell. Live chat must stay real-time. The nightly summaries and the tag backlog together cost twice as much as live chat, and nobody waits for them.
A job can use batch when both are true:
- no user is waiting for the result
- its deadline is at least the batch window, here 24 hours
The urgent reindex fails the second rule. It waits for nobody, and still cannot wait 24 hours.
Simple lesson: batch when nobody waits and the deadline allows it.
Write the Batch File
Batch APIs such as OpenAI's take a JSONL file: one JSON object per line. Each line has:
- a custom_id that you choose, to match the result later
- the method and url of the normal endpoint
- the normal request body: model, messages,
max_tokens
Results come back with the same custom_id, in any order.
Run the cell. The second prompt has quotes in it. A JSON library escapes them. Gluing strings together would produce a broken line.
The custom_id is your only link back. Make it unique and stable, such as job name plus ticket id. Many batch APIs limit its length. A duplicate id means two results you cannot tell apart.
Simple lesson: build lines with a JSON library, and make every id unique.
Merge Results and Retry Failures
A batch result file is not a clean list of answers. Look at the sample:
- the results are in a different order from the requests
- one line has an error object
- one line has a 429 status inside the response
- one request has no line at all
Your merge code must put answers back in request order, and collect everything that failed or is missing. You send those again, in the next batch or in real time.
Some batch requests fail. Rate limits, timeouts and server errors happen inside a batch too. The batch as a whole can say "completed" while some lines failed. Always check each line.
Run npm run batch (or dotnet run -- batch). The daily cost drops by 34%, and the merge reports 2 good results and 3 to retry.
Simple lesson: merge by id, never by position, and retry what failed.
Try it on a real model. Tag five real tickets, then price the same tokens at the real-time and the batch rate. Switch Model to Real in the lab's top bar. It runs an open model in your browser with WebLLM, and needs WebGPU. It is not graded.
Put It in Your App
- A job table. Store each batch job with its id, status, input file and output file. A scheduled worker polls the status.
- Idempotent merge. Merging the same result file twice must not create duplicate rows.
- Fallback. If a batch is late and the deadline is close, send the remaining items in real time.
- Cost records. Log batch calls with a
batch: trueflag and the batch price, so reports stay right.
Batch works with self-hosted models too. A local server has no batch discount, but running background work in large groups at night uses GPUs that would otherwise sit idle.
Simple lesson: batch needs a job table, a poller, a merge and a fallback.
Common mistakes
| Mistake | Wrong idea | Correct idea | What goes wrong |
|---|---|---|---|
| Batching anything that is not interactive | "Background means batch." | Check the deadline against the batch window. | Urgent work arrives late. |
| Building JSON lines by string concatenation | "It is just text." | Use a JSON library. | Quotes in a prompt break the file. |
| Matching results by position | "Results come back in order." | Match by custom_id. | Answers attach to the wrong tickets. |
| Trusting "completed" | "The batch finished, so all lines worked." | Check every line's error and status. | Failed items are silently lost. |
| No fallback | "Batches are never late." | Switch to real time near the deadline. | A missed morning report. |
Check Your Understanding
Exercises
Build It in the Project
The other files of this lesson's project are optional practice. Finish them to run the lesson's script on your own code.
What to Remember
- Batch APIs are cheaper and slower. Use them when nobody waits and the deadline allows.
- A batch file is JSONL: one request per line, each with a unique
custom_id. - Results come back in any order, and some lines fail.
- Merge by id, collect failures, and retry them.
- Keep a job table, a poller and a real-time fallback.
Next: Latency Engineering. You will cut the time the user waits by running independent calls together and streaming the reply.