Seeing and Reading in Multimodal AI

Lessons in this section

1. What Multimodal Models Can and Can't Do

   A model that can see is not a model that is right. Run ten claim items through a multimodal LLM and through specialist models, compare accuracy, cost and time, and learn four questions that pick the right tool.

2. Images In: Resolution, Crops and Token Cost

   Every pixel costs tokens, but shrinking a photo can erase the one number you need. Find the cheapest way to send twelve receipts that still reads every total, with resizing and crops.

3. Documents: OCR, Layout, Tables and Forms

   Invoices break extraction in four ways: merged cells shift columns, long tables get cut short, dates are read the wrong way round, and money stays as text. Fix each one, and extract ten invoices to a schema.

4. Screenshots, UI and Charts

   Turn error screenshots into complete bug reports and read charts honestly. Small error codes need OCR, summaries need a schema, and values read from bar heights must be flagged as estimates.

5. Verification and Grounding

   A model that sounds sure can still be wrong. Add cross-checks, format and range checks, grounding to a box on the page, and a confidence threshold, then route doubtful fields to a person.

6. Human Review and Your v1 Pipeline

   Build the review screen: each doubtful field beside its source region, one click to accept or correct, and every correction saved as an Evals row. Then export your v1 document pipeline.