# Documents: OCR, Layout, Tables and Forms

Course: [Multimodal AI](https://www.devtutorials.ai/learn/multimodal-ai) · Section: Seeing and Reading
Source: https://www.devtutorials.ai/learn/multimodal-ai/seeing-and-reading/documents-and-forms

> Invoices break extraction in four ways: merged cells shift columns, long tables get cut short, dates are read the wrong way round, and money stays as text. Fix each one, and extract ten invoices to a schema.

## What Document Extraction Is

**Document extraction** = turning a page (an invoice, a form, a bank letter) into typed data: dates, amounts, rows of a table.

In this lesson, you will extract ten supplier invoices into a schema, and fix the four ways that invoices break extraction.

A French supplier sends an invoice. On page 1, one description cell is two rows tall. The model reads the table as text, row by row. After the tall cell, a price lands in the quantity column.

Every value is real. One is in the wrong place. Nothing looks broken.

Press **Play**, or use **Next** to go one step at a time.

- **Frame 1:** the table as printed, with the merged cell.
- **Frame 2:** what the LLM returned: the second row has shifted.
- **Frame 3:** the fix: OCR's table cells sent with the page as hints.

**Simple lesson:** a table is a layout, not a line of text.

### A Real-World Example

Think about copying a timetable from a poster into a spreadsheet.

- You do not read it like a sentence. You read **across** each row and **down** each column.
- Where one cell covers two rows ("Weekends"), you know it applies to both.
- When the timetable continues on the next page, you keep adding rows in the same columns, and you skip the repeated header.

A model reading a page as text does none of this unless you help it:

- **Reading across and down** = layout hints: where each cell is.
- **The next page** = splitting the document into pages and joining the tables.
- **"Weekends" means Saturday and Sunday** = per-field rules: how to read each field.

**Simple lesson:** give the model the structure a person sees without thinking.

## What You Will Learn

By the end of this lesson, you will be able to:

- explain why tables and multi-page documents break extraction
- use layout hints from OCR to keep table columns in place
- split a long document into pages and join its tables
- use per-field rules for dates and money
- write a Structure stage that turns text into typed values

This lesson covers three stages: **Ingest** (splitting pages), **Perceive** (the model and its hints) and **Structure** (the schema).

## Four Ways Invoices Break

| Failure | What you see | Fix | Stage |
|---|---|---|---|
| Column shift | A price in the quantity column | Layout hints | Perceive |
| Cut-short table | Rows missing; the total is a page subtotal | Split pages, join tables | Ingest |
| Wrong date order | 03/04 read as 4 March | Per-field rules | Perceive |
| Text, not values | The total stored as "€2.418,60" | Schema | Structure |

You saw the column shift above. Here are the other three.

### Long Documents Get Cut Short

Step through all three frames.

- **Frame 1:** a 4-page invoice; the total is on page 4.
- **Frame 2:** sent as one file, the model stops after page 2.
- **Frame 3:** split into pages, read one by one, then joined.

**A missing page is a silent error.** The output looks complete: a list of rows and a total. Only a check against the page count shows that half of it is missing.

### Dates Read the Wrong Way Round

Step through both frames.

- **Frame 1:** 03/04/2026 from a UK supplier, read the US way.
- **Frame 2:** a per-field rule fixes every date.

### Text Is Not a Number

Step through both frames.

- **Frame 1:** a euro total uses a point for thousands and a comma for cents.
- **Frame 2:** without a schema, the text is stored; with one, it becomes a number.

**Simple lesson:** each failure has its own fix, in its own stage.

## The Schema Is the Contract

A **schema** says what each field must look like:

- `date`: a string like `2026-04-03` (YYYY-MM-DD)
- `total`: a number
- each line: a description, a number quantity and a number price

Structure turns the model's text into values that fit the schema. Anything that does not fit is an **error**, not data. It goes to a person (Lesson 5) instead of into your database.

Fix the failures one at a time and watch the score climb.

Step through all five frames.

- **The row of fixes:** green = on.
- **The ring:** key fields right out of 40.
- **The last frame:** every fix on, every field right.

Click the button and look at the **Pipeline** tab.

- **The invoice list:** ✓ or ✗ per invoice for the current settings.
- **The page:** click **Page 1, 2, …** for multi-page invoices.
- **The table on the right:** each typed field; open "What the model returned" to see the raw text.

**Easy way to remember:** layout for columns, pages for length, rules for formats, schema for types.

### Lab Exercise: Extract Ten Invoices to the Schema

Your task: change the **Ingest**, **Perceive** and **Structure** settings until the invoices pass. Then press **Check**.

Click the button and open the **Pipeline** tab.

- **Ingest:** split pages and join the tables.
- **Perceive:** pick the model, layout hints and per-field rules.
- **Structure:** validate against the schema.
- **To pass:** at least 95% of key fields right (invoice number, date, total, line items), and every multi-page table complete.

There is more than one way to pass. Compare the cost of the passing settings in the line under the table.

**Simple lesson:** most document accuracy is won in the stages around the model.

## Build It in TypeScript

The OCR + layout model returns every value as printed text. Your Structure stage turns that text into the schema's types.

```ts
function money(text: string, cur: "£" | "€"): number {
  const digits = text.replace(/[£€$\s]/g, "");
  // £1,234.50: comma for thousands.  €1.234,50: point for thousands, comma for cents.
  return Number(cur === "€" ? digits.replace(/\./g, "").replace(",", ".") : digits.replace(/,/g, ""));
}

export function toInvoice(raw: RawInvoice): Invoice {
  const [d, m, y] = raw.date.split("/"); // printed as DD/MM/YYYY
  return {
    number: raw.number,
    date: `${y}-${m}-${d}`,
    total: money(raw.total, raw.cur),
    lines: raw.lines.map(([desc, qty, unit]) => ({ desc, qty: Number(qty), unit: money(unit, raw.cur) })),
  };
}
```

**Check the result too.** In a real app, run the typed value through a schema library such as zod, so a bad value raises an error instead of quietly becoming `NaN`.

Click the button and open the **Code** tab.

- **The starter:** copies the text as it is. Rewrite it.
- **Run on the media bank:** your function runs on all 10 invoices' raw text.
- **To pass:** every invoice fully right, including the euro ones.

**Simple lesson:** Structure is your code, and it is where text becomes data.

## TL;DR

1. **Document extraction** = a page into typed data.
2. **Column shift** → send OCR's table cells as layout hints.
3. **Cut-short tables** → split into pages, read each, join the tables, skip repeated headers.
4. **Wrong date order** → a per-field rule: "dates are DD/MM/YYYY".
5. **Text, not values** → a schema, and a Structure stage that parses each field.
6. **A schema error** is not data. It goes to review.

## Common mistakes

| Mistake | Wrong idea | Correct idea | What goes wrong |
|---|---|---|---|
| Reading tables as text | "The model sees the table." | Send layout hints, or use a layout model. | Values in the wrong columns. |
| Sending a long PDF in one call | "It has the whole document." | Split, read each page, join. | Missing rows and a wrong total. |
| Trusting date order | "Dates are dates." | State the format for each field. | Every date up to the 12th can flip. |
| Storing text as money | "We can parse it later." | Parse in Structure, against a schema. | Sums break, or €2.418,60 becomes 2.4. |
| Counting the repeated header | "It is just another row." | Drop "Item (continued)" rows when joining. | Extra line items on every page. |

## Check Your Understanding

## Exercises

## What to Remember

- A table is a layout, not a line of text.
- Layout hints keep columns in place.
- Split long documents into pages, then join the tables.
- Tell the model the format of each field.
- Structure parses text into typed values, against a schema.
- A value that does not fit the schema goes to review, not to your database.

## Further Reading

- [Zod documentation](https://zod.dev/): a TypeScript schema library for validating extracted data.
- [Transformers.js documentation, by Hugging Face](https://huggingface.co/docs/transformers.js/index): OCR and document models that run in the browser.

Next: **Screenshots, UI and Charts**, where you turn error screenshots into complete bug reports and read charts honestly.