# JSON Validation

Course: [Large Language Models](https://www.devtutorials.ai/learn/llm) · Section: Using LLMs
Source: https://www.devtutorials.ai/learn/llm/using-llms/json-validation

> How to get JSON from an LLM and check it: parse it, check required keys and allowed values, clean common wrappers, and find the temperature where JSON starts to break.

Many apps do not need a chat reply. They need **data**: a product name, a severity level, a list of tags. The common way to get data from an LLM is to ask for **JSON**.

But the model writes JSON the same way it writes everything else: one likely token at a time. It can forget a quote, add a friendly sentence, or skip a key. So every JSON reply must be **checked** before your code uses it.

**The main idea:** ask for JSON, then always parse it and check it. Never trust it without looking.

## A Real-World Example

Think about a form that customers fill in by hand, like a delivery form.

Most forms come back fine. But some have a missing phone number, a date in the wrong box, or a note scribbled across the top. The warehouse does not ship a parcel from a form without checking it first. A clerk checks every form against a list: name filled in, address filled in, size is S, M, or L.

JSON from an LLM needs the same kind of clerk.

- **The paper form** is the model's JSON reply.
- **The list of required boxes** is the list of required keys.
- **"Size must be S, M, or L"** is a check on allowed values.
- **The clerk** is your validation code.
- **Sending a form back** is rejecting the reply, or asking the model again.

**In simple words:** treat model JSON like a hand-filled form. Check it before you use it.

## What You Will Learn

In this lesson, you will learn:

1. Why model JSON can break.
2. How to parse a reply safely with `JSON.parse`.
3. How to clean common wrappers, like code fences.
4. How to check required keys and allowed values.
5. How to grade a set of replies, like "7/10 passed".
6. How to find the temperature where the JSON starts to break.

The AI sandbox has a **JSON test** tab. It sends the same JSON task to the real model at several temperatures, checks every reply with `JSON.parse` and your required keys, and shows a pass or fail grid.

## Why JSON Breaks

JSON is strict. One wrong character makes the whole thing invalid. Here are the usual problems:

- **Extra text**, like "Sure! Here is the JSON:" before the object.
- **Code fences** around the JSON, like a markdown code block.
- **Missing quotes** around a key or a text value.
- **A missing key**, or a key with a different name.
- **A wrong value**, like `"urgent"` when only `low`, `medium`, or `high` are allowed.
- **A cut-off reply**, when max tokens ends the reply before the closing bracket.

Higher temperatures make these mistakes more likely, because unusual tokens get picked more often.

## Step 1: Parse Safely

`JSON.parse` turns a JSON string into an object. If the string is not valid JSON, it **throws an error**. Always wrap it in `try` and `catch`, so one bad reply does not crash your app.

```js runnable title="Parse safely with try and catch"
function safeParse(text) {
  try {
    return JSON.parse(text);
  } catch {
    return null;
  }
}

const good = '{"product": "Grid", "severity": "high"}';
const bad = '{"product": Grid, "severity": "high"}';

console.log("Good:", JSON.stringify(safeParse(good)));
console.log("Bad: ", safeParse(bad));
```

Expected output:

```text
Good: {"product":"Grid","severity":"high"}
Bad:  null
```

The bad reply is missing the quotes around `Grid`. `safeParse` returns `null` instead of crashing.

## Step 2: Clean Common Wrappers

Models often wrap JSON in a markdown code fence, or add a sentence before it. You can safely remove the **fence** and keep only the text from the first `{` to the last `}`.

```js runnable title="Remove code fences and extra text"
function extractJson(text) {
  const start = text.indexOf("{");
  const end = text.lastIndexOf("}");
  if (start === -1 || end === -1) return null;
  return text.slice(start, end + 1);
}

const fence = "`".repeat(3);
const reply = "Sure! Here is the JSON:\n" + fence + "json\n" + '{"product": "Chart", "severity": "low"}' + "\n" + fence;

const cleaned = extractJson(reply);
console.log(cleaned);
console.log(JSON.parse(cleaned).severity);
```

Expected output:

```text
{"product": "Chart", "severity": "low"}
low
```

Clean only what is safe to clean. Do not try to "repair" broken quotes or brackets by guessing. A broken reply should fail the check.

**In simple words:** trim the wrapping paper, but do not try to fix a broken parcel.

## Step 3: Check Keys and Values

Valid JSON is not enough. It must also have the **right keys**, and each value must be **allowed**. The check below returns a result in the same shape this course uses for all its graders: `passed`, and a `detail` that says exactly what is wrong.

```js runnable title="Check keys and allowed values"
function validate(obj) {
  if (!obj || typeof obj !== "object") return { passed: false, detail: "not a JSON object" };
  for (const key of ["product", "issue", "severity"]) {
    if (!(key in obj)) return { passed: false, detail: "missing key: " + key };
  }
  if (!["low", "medium", "high"].includes(obj.severity)) {
    return { passed: false, detail: "severity must be low, medium or high, got " + obj.severity };
  }
  return { passed: true, detail: "ok" };
}

console.log(validate({ product: "Grid", issue: "Export crash", severity: "high" }));
console.log(validate({ product: "Grid", severity: "high" }));
console.log(validate({ product: "Grid", issue: "Slow", severity: "urgent" }));
```

Expected output:

```text
{
  "passed": true,
  "detail": "ok"
}
{
  "passed": false,
  "detail": "missing key: issue"
}
{
  "passed": false,
  "detail": "severity must be low, medium or high, got urgent"
}
```

A good check never just says "wrong". It says which key or which value failed, so you can fix the prompt.

**Easy Way to Remember:** **parse, then check keys, then check values.** Three gates. A reply must pass all three.

## Step 4: Grade a Set of Replies

One good reply proves very little. Test the same prompt on many inputs, and count how many pass.

```js runnable title="Grade ten replies"
function safeParse(text) {
  try { return JSON.parse(text); } catch { return null; }
}

function passes(text) {
  const obj = safeParse(text);
  return !!obj && ["product", "severity"].every((k) => k in obj) && ["low", "medium", "high"].includes(obj.severity);
}

const replies = [
  '{"product":"Grid","severity":"high"}',
  '{"product":"Chart","severity":"low"}',
  'Here you go: {"product":"Scheduler","severity":"medium"}',
  '{"product":"Grid","severity":"urgent"}',
  '{"product":"PDF","severity":"low"}',
  '{"product":"Grid"}',
  '{"product":"Maps","severity":"medium"}',
  '{product:"Grid",severity:"high"}',
  '{"product":"Diagram","severity":"high"}',
  '{"product":"Kanban","severity":"low"}',
];

const passed = replies.filter(passes).length;
console.log(passed + "/" + replies.length + " cases passed");
```

Expected output:

```text
6/10 cases passed
```

Four replies failed:

- one has text before the JSON
- one has a severity that is not allowed
- one is missing a key
- one has keys without quotes

A score like "6/10" lets you compare prompts fairly: change the prompt, run the same ten cases, and see if the score goes up.

## Step 5: Find Where the JSON Breaks

How does temperature affect JSON? The **JSON test** tab in the sandbox measures this with the real model. The code below shows how to read such results. These numbers are an example of what a run might look like.

```js runnable title="At which temperature does the JSON break?"
const results = {
  0: [true, true, true],
  0.7: [true, true, true],
  1.2: [true, false, true],
  1.6: [false, false, true],
  2: [false, false, false],
};

const temperatures = Object.keys(results).map(Number).sort((a, b) => a - b);

for (const t of temperatures) {
  const ok = results[t].filter(Boolean).length;
  console.log("T = " + t + ":", ok + "/" + results[t].length, "valid");
}

const firstBreak = temperatures.find((t) => results[t].includes(false));

console.log("JSON starts to break at T =", firstBreak);
```

Expected output:

```text
T = 0: 3/3 valid
T = 0.7: 3/3 valid
T = 1.2: 2/3 valid
T = 1.6: 1/3 valid
T = 2: 0/3 valid
JSON starts to break at T = 1.2
```

The lesson from this: for JSON, use **temperature 0**. There is no benefit to randomness when you need a strict format.

## Try It in the AI Sandbox

Press the button. The sandbox opens on **JSON test** and runs your task at five temperatures, three times each. Each run uses a fixed seed, so the test repeats exactly.

Things to try:

- Find the lowest temperature where the first ✗ appears. Hover the cell to read the broken reply.
- Tick **JSON mode** and run again. The model is now forced to produce valid JSON. Do the keys still match?
- Remove "severity" from the task text, keep it in the required keys, and see the "missing" errors.

### JSON Mode

Many AI services, including WebLLM, have a **JSON mode**. It limits the model so it can only pick tokens that keep the output valid JSON. That fixes **parsing** errors. It does not guarantee the right **keys** or **values**, so you still need Step 3.

```js
// WebLLM, and other OpenAI-style APIs
const reply = await engine.chat.completions.create({
  messages,
  temperature: 0,
  response_format: { type: "json_object" },
});
```

Never pass model JSON straight into a database, a payment, or another system without validation. Even with JSON mode and temperature 0, a model can choose a wrong value, invent a field, or misread the input. Validate every reply, every time.

## Common mistakes

| Mistake | Wrong idea | Correct idea | What goes wrong |
|---|---|---|---|
| Parsing without try/catch | "The model always returns JSON." | One bad reply throws an error. | The app crashes on a single broken reply. |
| Valid JSON means correct data | "It parsed, so it is right." | Keys and values must be checked too. | Missing fields or wrong values reach your database. |
| High temperature for JSON | "A bit of randomness is harmless." | Unusual tokens break strict formats. | More replies fail to parse. |
| Max tokens too low | "Short replies are cheaper." | The reply can stop before the closing bracket. | Long objects are always invalid. |
| Testing one case | "It worked once, so the prompt is fine." | Test many cases and count passes. | The prompt fails on inputs you did not try. |
| Trusting JSON mode fully | "JSON mode fixes everything." | It fixes parsing, not keys or values. | Valid JSON with the wrong fields slips through. |

## Exercises

Each exercise ends with a value on the last line. Press **Check** to compare it with the expected answer.

## What to Remember

- Model JSON can break: extra text, fences, missing quotes, missing keys, wrong values, cut-off replies.
- Always **parse inside try and catch**.
- Clean only safe wrappers, like **code fences**. Never guess fixes.
- After parsing, **check keys**, then **check values**.
- Give clear failure details, like **"missing key: issue"**.
- Grade prompts on **many cases**, like **"7/10 passed"**.
- Use **temperature 0** for JSON. **JSON mode** helps with parsing, but you still validate.

Next: **LLM APIs**, how to call a model from your own code.