# Build vs Buy: Open-Weight, Self-Host or API Course: [LLM Cost, Latency & Model Selection](https://www.devtutorials.ai/learn/llm-cost-latency-model-selection) · Section: Operate at Scale Source: https://www.devtutorials.ai/learn/llm-cost-latency-model-selection/operate-at-scale/build-vs-buy-open-weight-self-host-or-api > Decide when running an open model yourself starts to pay: size GPUs for the peak, compute the total cost of ownership with people and fixed costs, and find the crossover volume, in TypeScript or C#. ## Should We Run the Model Ourselves? Every model in this course is an open model of 7B or smaller. You could rent GPUs and run them yourself, with a server such as vLLM. Someone always asks: "GPUs are a few dollars an hour. The API bill is thousands a month. Why are we paying per token?" The answer depends on volume, and on costs that are easy to leave out: - GPUs cost money every hour, busy or idle - you must size for the busy hour, not the average - you need at least two GPUs to survive one failing - someone has to run it: upgrades, on-call, security reviews Press **Play**, or use **Next** to go one step at a time. - **Each frame:** one monthly volume, and what each option costs. - **Green:** the cheapest option at that volume. **Simple lesson:** per-token APIs win at low volume, owned GPUs win at high volume, and the crossover is a number you can compute. ### A Real-World Example A team moved to self-hosting at 10 billion tokens a month. The GPU bill was lower than the API bill. The total was higher: two engineers spent days a month on upgrades, outages and a security review. Their spreadsheet had a GPU line and no people line. **Simple lesson:** total cost of ownership includes the people. ## What You Will Build In this lesson, you will learn: 1. how to size GPUs for the peak, with headroom and a minimum 2. how to compute the monthly total cost of each option 3. how to find the crossover volume, and how much it depends on assumptions You will finish three files in the **tco** folder: - **Capacity** (`src/tco/capacity.ts`, or `Tco/Capacity.cs`): GPUs needed. - **TCO** (`src/tco/tco.ts`, or `Tco/Tco.cs`): monthly cost per option. - **Crossover** (`src/tco/crossover.ts`, or `Tco/Crossover.cs`): the cheapest option per volume. Open the **Code** tab. The harness has three options for Qwen2.5 7B, with every assumption as a field you can change: - **Hosted API:** pay per token. - **Dedicated endpoint:** rented GPUs that a vendor runs for you. - **Self-hosted:** your own GPUs and your own team. **These numbers are a model, not a quote.** GPU prices, throughput and staff costs vary a lot. The value of the code is that you can put in your own numbers and see where the answer changes. **Simple lesson:** write the decision as code with named assumptions. ## Size for the Peak GPUs are sized by tokens per second: - **average** tokens per second = tokens per month ÷ seconds in a month - **peak** = average × a peak factor, because traffic is not flat - each GPU runs at a **target load**, such as 60%, so it has headroom - round **up**, and never go below the **minimum**, such as 2 for one spare **Simple lesson:** size for the busy hour, with a spare. ## The Total Cost of Ownership For each option, per month: - **API:** tokens ÷ 1,000,000 × the price per million tokens - **GPU options:** GPUs × hourly price × 730 hours, plus ops hours × hourly rate, plus fixed costs such as monitoring and compliance The API has no fixed cost. The GPU options have a large fixed cost and a small cost per extra token. **Simple lesson:** GPUs and people are fixed costs, and per-token prices are not. ## Find the Crossover Run the cell. - **Up to 25B tokens a month:** the API is cheapest. - **At 50B tokens a month:** self-hosting becomes cheapest. - **At 200B:** self-hosting costs about 40% less than the API. Run the cell. Each line changes one assumption. - **Faster GPUs** move the crossover down. - **More ops hours** keep it where it is in this table, and change which GPU option wins. - **A cheaper API** pushes it out of range. **Recheck the decision when prices move.** API prices for small models tend to fall. A crossover that looked close last year may be gone today. Re-run this with current prices before you commit to hardware. Run `npm run tco` (or `dotnet run -- tco`). It prints the GPUs needed and the cheapest option per volume, and finds the crossover at 50B tokens a month. **Simple lesson:** compute the crossover, and see how far your assumptions can move it. ## When Does Fine-Tuning Pay? Fine-tuning a small model can replace a long prompt and a big model. It is a build decision too: - **one-off cost:** labelling data, training runs and the eval - **monthly cost:** hosting the fine-tuned model, and the people who run it - **saving:** the tokens and the bigger model you no longer pay for Fine-tune only after the eval shows the small model is as good on your golden set. Run the cell. At 1M requests a month it never pays back. At 5M it pays back in under a month. **Simple lesson:** fine-tuning is fixed cost for variable savings, so it pays only at volume. **Try it on a real model.** Measure your machine's real output speed. It turns the throughput assumption into a number you measured. Switch **Model** to **Real** in the lab's top bar. It runs an open model in your browser with WebLLM, and needs WebGPU. It is not graded. ## Put It in Your App - **Get your volume from records.** Sum input and output tokens per month from your cost records, and find the busy-hour peak from them too. - **Price fine-tuning the same way.** A fine-tuned small model is a self-hosting decision with a training cost on top. It pays only at volume, and only if the eval says it is as good. - **Keep the API path.** Even when you self-host, keep the API as a fallback for outages and spikes. **Try serving a model locally first.** Run one of the course's open models with Ollama on your own machine, and measure its tokens per second. It is a cheap way to make the throughput assumption real. **Simple lesson:** decide from your own volume and peak, and keep a fallback. **For product managers:** self-hosting is a commitment of people and time, not only GPUs. Ask for the crossover volume and the team cost before you agree to it. ## Common mistakes | Mistake | Wrong idea | Correct idea | What goes wrong | |---|---|---|---| | Sizing for the average | "We need average throughput." | Size for the peak, with headroom. | Slow replies in the busy hour. | | One GPU | "One is enough at our volume." | Keep a minimum for failover. | One failure is a full outage. | | No people cost | "Hardware is the cost." | Add ops hours and compliance. | Self-hosting looks cheaper than it is. | | Comparing at one volume | "We checked our volume today." | Compare across volumes. | Growth flips the answer. | | No API fallback | "We own the model now." | Keep the API for spikes and outages. | Every GPU problem is user-facing. | ## Check Your Understanding ## Exercises ## Build It in the Project The other files of this lesson's project are optional practice. Finish them to run the lesson's script on your own code. ## What to Remember - Size GPUs for the peak, with headroom and a minimum for failover. - Total cost of ownership includes GPUs, people and fixed costs. - The API wins at low volume, owned GPUs at high volume. - Compute the crossover, and test how assumptions move it. - Keep the API as a fallback when you self-host. Next: **Capstone: A Cost-and-Latency Kit for Your Feature**. You will apply every lever to the expensive agent, one at a time, and export the kit.