# Structured Output and Validation

Course: [Building AI Agents](https://www.devtutorials.ai/learn/building-ai-agents) · Section: One Agent That Works
Source: https://www.devtutorials.ai/learn/building-ai-agents/basics/checking-the-answer

> Give the helper's answer a fixed shape with named boxes, then check every box before the customer sees it. Send problems back for a fix, and pass to a person after too many tries.

## Structured Output and Validation

Ravi writes to the shop: "Where is my phone case?"

He gives no order number, so Shop Helper has nothing to look up.

It answers anyway, and it sounds sure:

"Order 9999 (your item) is on its way. Next step: wait a few more days."

There is no order 9999. No tool ever returned it. The helper made it up.

This lesson teaches how to give the answer a fixed shape. Then you check it before the customer sees it.

**Simple lesson:** never send an answer you have not checked.

### A Real-World Example

Think about a parcel desk at the post office.

- You fill in a form with set boxes: name, address, weight.
- The clerk checks each box before taking the parcel.
- An empty box? The clerk hands the form back to you.
- A wrong postcode? You fix that one box and try again.
- Still wrong after two tries? A manager comes to help.

Your AI helper can work the same way. The model fills the form. The app is the clerk.

**Simple lesson:** a form with boxes can be checked one box at a time.

## What You Will Learn

In this lesson, you will learn:

1. Why free text is hard to check, and what an answer form is.
2. The three checks: empty box, word not allowed, number not found.
3. How a made-up order number gets caught.
4. How repair and retry fixes a bad box.
5. When to stop trying and pass the chat to a person.

## Free Text or a Form

So far, Shop Helper has answered in free text. That means one sentence, in any words the model likes.

Free text reads well. But the app cannot check it, because it cannot tell which words are the status or the date.

**Answer form** = a fixed shape for the reply, with named boxes the model must fill. Real tools call this an output schema.

Here is the answer form for an order question:

```text
Answer form: order question
  order number   must match an order in a tool result
  item           words
  status         only: packing, shipped, delivered
  date           arrives on, or ships on while packing
  next step      words
```

Each box holds one fact. Code in the app can read each box by its name.

The rules on the right are part of the form. They tell the model what each box may hold. They also tell the check what to test.

The date box means "arrives on" for a shipped order. For an order still being packed, it means "ships on". So order 1043 shows Friday 31 Oct as its ship date.

Many AI tools can make the model reply in a fixed shape. In this lab, the pretend model always fills the boxes, but it can still fill them badly.

Press **Play** and compare the two answers.

- **Free text:** it reads well, but nothing in it can be checked.
- **The form:** the same facts, each one in its own named box.
- **The check:** each box turns green when it passes.
- **The end:** the app turns the form back into friendly words for the customer.

**Simple lesson:** a form lets the app check each fact, and the customer still reads normal words.

## The Check

The form gives the app something to test. Now the app needs rules.

**Check** = code that tests every box of the answer form before the customer sees it. Real tools call this validation.

In this lesson, the check has three rules:

- **No empty box.** Every box must have something in it.
- **Only allowed words.** The status box takes only packing, shipped or delivered.
- **No made-up numbers.** Every order number must appear in a tool result.

The check returns a list of problems in plain words. An empty list means the form passes.

The check does not need an AI model. It is a few lines of plain code, so it is fast and costs no coins.

Notice what the check does not test. "Shipped" is an allowed word, even when the order is still being packed. So a good app also compares key boxes with the tool result. You will do that in the first exercise.

**Why only three words for status?**

Other parts of the app read the status box. A tracking page may show a picture for each status.

That code knows three words. A friendly phrase like "on its way" would break it, because no picture matches.

### The Schema Guards the Shape

Many model APIs can force the reply to match a form. This is often called **structured output** with a schema.

So the API may already enforce required boxes and allowed words. The model cannot leave the status box empty or write "on its way".

Your own check is still needed for the facts:

- **Is order 9999 real?** The schema only sees four digits.
- **Does the date match the tool result?** "Thursday 30 Oct" is a real date. But the tool said Friday 31 Oct.

The schema guards the shape. Your check guards the facts.

**Simple lesson:** the check is plain code, so it gives the same result every time.

## Catching a Made-Up Number

Back to Ravi. He gave no order number, so no tool ran.

The pretend model guesses when facts are missing. It filled every box with a guess.

Press **Play** and watch the order number box.

- **The guess:** the form looks complete and confident.
- **The stamp:** the check cannot find order 9999 in any tool result.
- **More problems:** the status word is not allowed, and one box is empty.
- **Not sent:** Ravi never sees the made-up number.

**A made-up number is worse than no answer.**

Ravi might wait days for an order that does not exist. He might also share the number with your support team, who will find nothing.

Always check numbers against what the tools really returned.

**Simple lesson:** if a tool did not return it, the answer may not say it.

## Repair and Retry

A failed check does not have to end the chat. Often the model can fix the problem.

The customer does not see any of this. They only see the answer that finally passes.

**Repair and retry** = the app sends the problems back to the model, and the model fills the form again.

The pretend model follows one rule here. When a problem names a box, it fixes that box from the tool results. It keeps every other box as it was.

Watch an order question where one box is left empty.

Press **Play** and follow the form through the check.

- **Try 1:** the customer did not ask when, so the model skips the date box.
- **The way out:** the check fails, so the form goes back to the model.
- **Try 2:** the problem names the date box. The model fills it from the tool result.
- **Sent:** every box passes, so the customer gets the reply.

Now watch a word that is not allowed.

Press **Play** and watch the status box.

- **Try 1:** the model writes "on its way", which sounds friendly.
- **The check:** that phrase is not one of the three allowed words.
- **Try 2:** the model reads the tool result again and writes "shipped".
- **Green:** the fixed box is marked, so you can see what changed.

**Write problems the model can act on**

- Name the box: "The date box is empty."
- Say what is allowed: "Use packing, shipped or delivered."
- Say what was not found: "Order 9999 is not in any tool result."

A clear problem gets fixed on the next try. A vague one, like "invalid", often does not.

**Simple lesson:** send the problem back in plain words, and the model can usually fix it.

## When to Stop Trying

Repair and retry has a limit. Each try is one more model call, so it costs time and money.

Every repair and retry also counts toward the step limit and the budget limit, as in **The Agent Loop by Hand**.

Some problems cannot be fixed by trying again. If the facts are missing, the model has nothing to fix the box with.

So the app sets a limit on tries. In this lab, the limit is 2.

Real apps often use a small limit, such as 2 or 3. A higher limit rarely helps, because the same missing fact stays missing.

After each check, the app makes one of three choices:

- **Send it** when every box passes.
- **Send it back to fix** when a box fails and tries are left.
- **Pass to a person** when a box still fails and no tries are left.

Press **Play** and watch the tries counter.

- **Try 1:** no order number, so every fact is a guess. The check sends it back.
- **Try 2:** the model drops its guesses, but now those boxes are empty.
- **The limit:** two tries are used up, so a person takes the chat.
- **Honest message:** the customer is told a person will reply. Nothing made up is sent.

**Simple lesson:** set a limit on tries, and pass to a person when it runs out.

## Where This Is Used

Checked answers show up in many apps:

- **Support helpers** check order numbers and statuses before replying.
- **Form-filling copilots** check that dates and amounts are in the right format.
- **Report helpers** check that every number came from the real data.

Many teams log each failed check. The log shows which boxes fail most, so they can fix the prompt or the tool.

**Simple lesson:** wherever an answer feeds other code or a customer, check it first.

## In Code

This code builds the answer form, the check and the repair loop, with the same Shop Helper orders.

```ts runnable title="Check the answer form, repair, then pass to a person"
type Status = "packing" | "shipped" | "delivered";
interface AnswerForm {
  orderId: string;
  item: string;
  status: string;
  date: string;
  nextStep: string;
}
interface ToolResult {
  orderId: string;
  item: string;
  status: Status;
  date: string;
}

const ALLOWED: string[] = ["packing", "shipped", "delivered"];

// The check: plain code, no model call.
function validate(form: AnswerForm, results: ToolResult[]): string[] {
  const problems: string[] = [];
  const boxes: (keyof AnswerForm)[] =
    ["orderId", "item", "status", "date", "nextStep"];
  for (const box of boxes) {
    if (form[box].trim() === "") problems.push(`The ${box} box is empty.`);
  }
  if (form.status !== "" && ALLOWED.indexOf(form.status) < 0) {
    const allowed = "Use packing, shipped or delivered.";
    problems.push(`Status "${form.status}" is not allowed. ${allowed}`);
  }
  const known = results.some((r: ToolResult) => r.orderId === form.orderId);
  if (form.orderId !== "" && !known) {
    problems.push(`Order ${form.orderId} is not in any tool result.`);
  }
  return problems;
}

// A scripted fake model: try 1 is the first form, then it fixes named boxes.
function fakeModel(
  tryNo: number, problems: string[], last: AnswerForm, results: ToolResult[],
): AnswerForm {
  if (tryNo === 1) return last;
  const fixed: AnswerForm = { ...last };
  const r = results[0];
  if (r === undefined) return { ...fixed, orderId: "", status: "", date: "" };
  for (const p of problems) {
    if (p.indexOf("date") >= 0) fixed.date = r.date;
    if (p.indexOf("Status") >= 0) fixed.status = r.status;
  }
  return fixed;
}

function answer(label: string, first: AnswerForm, results: ToolResult[]): void {
  console.log(label);
  let form = first;
  let problems: string[] = [];
  for (let tryNo = 1; tryNo <= 2; tryNo++) {
    form = fakeModel(tryNo, problems, form, results);
    problems = validate(form, results);
    console.log(`  try ${tryNo}: ${problems.length === 0 ? "passes" : "problems"}`);
    for (const p of problems) console.log(`    - ${p}`);
    if (problems.length === 0) {
      console.log(`  decision: send it (${form.orderId}, ${form.status}, ${form.date})`);
      return;
    }
  }
  console.log("  decision: pass to a person");
}

const lookup1044: ToolResult[] = [
  { orderId: "1044", item: "rain jacket", status: "shipped", date: "Monday 3 Nov" },
];
answer("Asha asks about 1044:", {
  orderId: "1044", item: "rain jacket", status: "on its way", date: "",
  nextStep: "Track it on the order page.",
}, lookup1044);

answer("Ravi gives no number:", {
  orderId: "9999", item: "lamp", status: "on its way", date: "Friday",
  nextStep: "Wait for it.",
}, []);
```

Expected output:

```text
Asha asks about 1044:
  try 1: problems
    - The date box is empty.
    - Status "on its way" is not allowed. Use packing, shipped or delivered.
  try 2: passes
  decision: send it (1044, shipped, Monday 3 Nov)
Ravi gives no number:
  try 1: problems
    - Status "on its way" is not allowed. Use packing, shipped or delivered.
    - Order 9999 is not in any tool result.
  try 2: problems
    - The orderId box is empty.
    - The status box is empty.
    - The date box is empty.
  decision: pass to a person
```

- **The form:** `AnswerForm` has one named box per fact. `ToolResult` is what the lookup really returned.
- **The check:** `validate` returns problems in plain words. It looks for empty boxes, words not allowed and order numbers no tool returned.
- **The fake model:** on try 2 it fixes only the boxes the problems name, from the tool result. With no tool result, it drops its guesses.
- **The limit:** after 2 tries the app stops and passes the chat to a person.

### In a Real App

Real apps often write the answer form with zod and check it with `safeParse`. This sketch is simplified, and `ModelClient` is a small interface you would back with your provider's SDK.

```ts
import * as z from "zod";

const AnswerForm = z.object({
  orderId: z.string().regex(/^\d{4}$/),
  item: z.string().min(1),
  status: z.enum(["packing", "shipped", "delivered"]),
  date: z.string().min(1),
  nextStep: z.string().min(1),
});
type AnswerForm = z.infer<typeof AnswerForm>;

interface ModelClient {
  fillForm(problems: string[]): Promise<unknown>; // returns parsed JSON
}

async function checkedAnswer(
  model: ModelClient,
  toolResults: { orderId: string }[],
) {
  let problems: string[] = [];
  for (let tryNo = 1; tryNo <= 2; tryNo++) {
    const parsed = AnswerForm.safeParse(await model.fillForm(problems));
    if (!parsed.success) {
      // The shape is wrong: name each bad box.
      problems = parsed.error.issues.map(
        (i) => `${i.path.join(".")}: ${i.message}`,
      );
      continue;
    }
    // The shape is right. Now check the facts against the tool results.
    const form: AnswerForm = parsed.data;
    if (!toolResults.some((r) => r.orderId === form.orderId)) {
      problems = [`Order ${form.orderId} is not in any tool result.`];
      continue;
    }
    return { send: form };
  }
  return { passToPerson: problems };
}
```

Many model APIs can also enforce a JSON Schema on the model's output, often called structured outputs. That guards the shape, but not the facts, so the check against tool results stays.

**Simple lesson:** the schema check and the fact check decide: send it, fix it, or pass it to a person.

## TL;DR

1. **Answer form** = a fixed shape for the reply, with named boxes. Also called an output schema.
2. **Check** = code that tests every box before the customer sees it. Also called validation.
3. **The three checks** = no empty box, only allowed words, no number that a tool did not return.
4. **Repair and retry** = the app sends problems back, and the model fills the form again.
5. **The limit** = after a set number of failed tries, pass the chat to a person.

## Common mistakes

| Mistake | Wrong idea | Correct idea | What goes wrong |
|---|---|---|---|
| Sending free text straight to the customer | "The reply reads well, so it is right." | Ask for an answer form and check it first. | A made-up order number reaches the customer. |
| Checking only that boxes are filled | "If every box has something, it is fine." | Also check allowed words and numbers against tool results. | "Order 9999" passes because the box is not empty. |
| Vague problem messages | "Just say the answer is invalid." | Name the box and say what is allowed. | The model cannot tell what to fix, so it fails again. |
| No limit on tries | "Keep retrying until it passes." | Set a limit, then pass to a person. | The loop never ends, and the cost keeps growing. |
| Retrying when facts are missing | "One more try will fix it." | If no tool returned the fact, a person must ask for it. | Each try guesses again or leaves the box empty. |

## Check Your Understanding

## Exercises

## What to Remember

- Free text reads well, but the app cannot check it.
- An **answer form** puts each fact in its own named box.
- The **check** tests every box: not empty, allowed words only, no made-up numbers.
- **Repair and retry** sends clear problems back so the model can fix them.
- Set a limit on tries, then pass the chat to a person.

Next: **State, Memory and Context Limits**, how the helper keeps important facts when a chat gets long.