# LLM APIs

Course: [Large Language Models](https://www.devtutorials.ai/learn/llm) · Section: Using LLMs
Source: https://www.devtutorials.ai/learn/llm/using-llms/llm-apis

> How code talks to an LLM: the chat messages format, request settings, streaming replies, token usage and cost, errors and retries, with WebLLM and cloud services.

So far, you have used a model through the AI sandbox. Apps do the same thing in code: they send a **request** to a model and read the **response**. The code that does this is called an **LLM API**.

Good news: almost every LLM API today uses the same shape, first made popular by OpenAI's "chat completions". WebLLM, the library behind the sandbox, uses it too. Learn it once, and you can use most models.

**The main idea:** you send a list of messages plus a few settings, and you get back the reply plus how many tokens were used.

## A Real-World Example

Think about ordering food through a delivery app.

You fill in an order form: what you want, any special notes, how spicy, and the address. You press Send. The kitchen sends back the food and a receipt. If the kitchen is busy, the app tells you to wait and tries again.

Calling an LLM works the same way.

- **The order form** is the request: messages and settings.
- **Special notes** are the system message.
- **"How spicy"** is a setting, like temperature.
- **The food** is the reply text.
- **The receipt** is the token usage, which decides the cost.
- **"Busy, try again"** is an error with a retry.

**In simple words:** an LLM API is an order form for text.

## What You Will Learn

In this lesson, you will learn:

1. The parts of a chat completion request.
2. How to read the response and its token usage.
3. How to keep a conversation going.
4. How streaming sends the reply in pieces.
5. How to work out the cost of a request.
6. How to handle errors with retries.
7. How the same code works with WebLLM and with cloud services.

The code boxes on this page use small stand-in objects, so they run anywhere without a model or an API key. The plain code blocks show the real WebLLM and cloud calls. The **Chat** tab in the AI sandbox runs exactly these calls in your browser.

## The Request

A chat completion request has two parts: the **messages** and the **settings**.

```js runnable title="Build a request"
const request = {
  model: "Qwen2.5-0.5B-Instruct-q4f16_1-MLC",
  messages: [
    { role: "system", content: "You are a support assistant. Answer in one sentence." },
    { role: "user", content: "How do I reset my password?" },
  ],
  temperature: 0.3,
  top_p: 0.9,
  max_tokens: 100,
};

console.log("Model:", request.model);
console.log("Messages:", request.messages.length);
console.log("Settings:", JSON.stringify({ temperature: request.temperature, top_p: request.top_p, max_tokens: request.max_tokens }));
```

Expected output:

```text
Model: Qwen2.5-0.5B-Instruct-q4f16_1-MLC
Messages: 2
Settings: {"temperature":0.3,"top_p":0.9,"max_tokens":100}
```

You know every part from earlier lessons:

- **model** says which model to run.
- **messages** is the chat, with **system**, **user**, and **assistant** roles.
- **temperature** and **top_p** control sampling.
- **max_tokens** limits the length of the reply.

Other common settings are **seed** for repeatable output, **stop** for extra stop words, and **response_format** for JSON mode.

## The Response

The response has the reply, the reason it stopped, and the token **usage**.

```js runnable title="Read a response"
const response = {
  choices: [
    {
      message: { role: "assistant", content: "Click 'Forgot password' on the sign-in page and follow the email link." },
      finish_reason: "stop",
    },
  ],
  usage: { prompt_tokens: 38, completion_tokens: 17, total_tokens: 55 },
};

const choice = response.choices[0];
console.log("Reply:", choice.message.content);
console.log("Stopped because:", choice.finish_reason);
console.log("Tokens used:", response.usage.total_tokens);
```

Expected output:

```text
Reply: Click 'Forgot password' on the sign-in page and follow the email link.
Stopped because: stop
Tokens used: 55
```

- **choices[0].message.content** is the reply text.
- **finish_reason** is `"stop"` when the model chose its end token, and `"length"` when it hit **max_tokens**.
- **usage** counts the prompt tokens and the reply tokens.

**In simple words:** read the text from `choices[0]`, and the cost from `usage`.

## Keeping a Conversation Going

The model has no memory, as you saw in Generation Flow. To keep a chat going, your code adds the assistant's reply to the message list, then the next user message, and sends the **whole list** again.

```js runnable title="Add each turn to the history"
const messages = [{ role: "system", content: "Be brief." }];

function addTurn(question, reply) {
  messages.push({ role: "user", content: question });
  messages.push({ role: "assistant", content: reply });
}

addTurn("What is an LLM?", "A model that predicts text.");
addTurn("Name one.", "Qwen2.5.");

messages.push({ role: "user", content: "How big is it?" });
console.log("Messages sent on turn 3:", messages.length);
console.log(messages.map((m) => m.role).join(" -> "));
```

Expected output:

```text
Messages sent on turn 3: 6
system -> user -> assistant -> user -> assistant -> user
```

This is also how the **Clear chat** button in the sandbox works: it simply empties this list.

## Streaming

With **streaming**, the reply arrives in small pieces called **chunks**, one or a few tokens at a time. Your code joins them as they arrive and shows the text so far.

```js runnable title="Join streamed chunks"
const chunks = [
  { choices: [{ delta: { content: "Click" } }] },
  { choices: [{ delta: { content: " 'Forgot" } }] },
  { choices: [{ delta: { content: " password'" } }] },
  { choices: [{ delta: { content: " on the sign-in page." } }] },
  { choices: [{ delta: {}, finish_reason: "stop" }] },
];

let text = "";
for (const chunk of chunks) {
  const piece = chunk.choices[0].delta.content;
  if (piece) {
    text += piece;
    console.log("So far:", text);
  }
}
```

Expected output:

```text
So far: Click
So far: Click 'Forgot
So far: Click 'Forgot password'
So far: Click 'Forgot password' on the sign-in page.
```

In a stream, each chunk has a **delta**, meaning "the new part". The last chunk has the **finish_reason**.

## Cost From Usage

Paid services charge per token, usually with different prices for input and output. The usage numbers let you work out the cost of every request. The prices below are made up for the example.

```js runnable title="Cost of one request and of a day"
const usage = { prompt_tokens: 1200, completion_tokens: 300 };
const pricePerMillion = { input: 0.5, output: 1.5 };  // example prices in dollars

const cost =
  (usage.prompt_tokens / 1000000) * pricePerMillion.input +
  (usage.completion_tokens / 1000000) * pricePerMillion.output;

console.log("One request: $" + cost.toFixed(5));
console.log("5,000 requests a day: $" + (cost * 5000).toFixed(2));
```

Expected output:

```text
One request: $0.00105
5,000 requests a day: $5.25
```

Models running in the browser, like the one in the sandbox, cost nothing per token. They use the reader's own computer instead.

## Errors and Retries

Requests can fail. A cloud service may be busy, or rate-limit you for sending too many requests too fast. A browser model may fail to load. Good code **retries** a few times, waiting a little longer each time. This is called **backoff**.

```js runnable title="Retry with growing waits"
const outcomes = ["busy", "busy", "ok"];  // what the service answers on each try

function callWithRetry(maxTries) {
  let wait = 500;
  for (let attempt = 1; attempt <= maxTries; attempt++) {
    const result = outcomes[attempt - 1];
    if (result === "ok") return "Success on try " + attempt;
    console.log("Try " + attempt + ": " + result + ", waiting " + wait + " ms");
    wait *= 2;
  }
  return "Gave up after " + maxTries + " tries";
}

console.log(callWithRetry(4));
```

Expected output:

```text
Try 1: busy, waiting 500 ms
Try 2: busy, waiting 1000 ms
Success on try 3
```

Retry only errors that may go away, like "busy" or a network drop. Do not retry a request that is wrong, like a missing message. It will fail every time.

**Easy Way to Remember:** **send messages and settings, read choices and usage, stream deltas, retry with backoff.**

## The Real Calls

### WebLLM, in the Browser

This is the code the AI sandbox runs, in short. The model downloads once and runs on the reader's graphics chip.

```js
import { CreateMLCEngine } from "@mlc-ai/web-llm";

const engine = await CreateMLCEngine("Qwen2.5-0.5B-Instruct-q4f16_1-MLC", {
  initProgressCallback: (p) => console.log(p.text),
});

const reply = await engine.chat.completions.create({
  messages: [{ role: "user", content: "Say hello in French." }],
  temperature: 0,
});
console.log(reply.choices[0].message.content);
```

### Streaming With WebLLM

```js
const stream = await engine.chat.completions.create({
  messages: [{ role: "user", content: "Write a haiku about tea." }],
  stream: true,
});

let text = "";
for await (const chunk of stream) {
  text += chunk.choices[0]?.delta?.content ?? "";
}
console.log(text);
```

### A Cloud Service

A cloud service uses the same request shape, sent over HTTP. The address and the model name depend on the provider.

```js
const response = await fetch("https://api.example.com/v1/chat/completions", {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    Authorization: "Bearer " + process.env.LLM_API_KEY,
  },
  body: JSON.stringify({
    model: "provider-model-name",
    messages: [{ role: "user", content: "Say hello in French." }],
    temperature: 0,
  }),
});
const data = await response.json();
console.log(data.choices[0].message.content);
```

Many providers follow this shape closely, but not all of them. Some use their own field names. Always check the provider's documentation.

Never put a cloud API key in code that runs in the browser. Anyone can open the page source and copy it, then run up your bill. Call cloud models from your own server, and keep the key there. Browser models like WebLLM need no key at all.

## Try It in the AI Sandbox

Press the button to send a message from the **Chat** tab. After the reply, look at the line under it: the token count and the speed come from the response's **usage**. Press **Stop** during a long reply to see an interrupted stream.

## Common mistakes

| Mistake | Wrong idea | Correct idea | What goes wrong |
|---|---|---|---|
| Sending only the new message | "The model remembers the chat." | Send the full message list each time. | The model forgets everything said before. |
| Ignoring finish_reason | "Every reply is complete." | `"length"` means the reply was cut at max_tokens. | Half-finished answers and broken JSON reach users. |
| API key in browser code | "Nobody will look." | Page code is public. Keep keys on a server. | Someone copies the key and spends your money. |
| Retrying everything | "Just try again until it works." | Retry only errors that can go away, with waits. | Endless retries on a bad request, or a ban for flooding. |
| Not tracking usage | "Tokens are cheap." | Costs add up with long histories and many users. | A surprise bill at the end of the month. |
| Assuming identical APIs | "All providers are the same." | Most share the shape, but details differ. | Code breaks when you switch providers. |

## Exercises

Each exercise ends with a value on the last line. Press **Check** to compare it with the expected answer.

## What to Remember

- Most LLM APIs share the **chat completions** shape.
- A **request** has **messages** and **settings**: temperature, top_p, max_tokens, seed, response_format.
- The **response** has **choices[0].message.content**, a **finish_reason**, and **usage**.
- Send the **whole history** every turn. The model does not remember.
- **Streaming** sends **deltas**. Join them as they arrive.
- Work out **cost from usage**. Retry **busy** errors with **backoff**.
- Keep **API keys on a server**, never in the browser.

You have finished the LLM course. You now know how a language model reads text, predicts the next token, picks it, and how to use it safely from code. The **Prompt Engineering** course builds on all of this.