# Multimodal Search

Course: [Multimodal AI](https://www.devtutorials.ai/learn/multimodal-ai) · Section: Listening, Speaking and Generating
Source: https://www.devtutorials.ai/learn/multimodal-ai/listening-and-generating/multimodal-search

> Search a document archive by words and by looks: OCR text and page-image embeddings, routing, re-ranking, and answers that cite their pages.

## What Multimodal Search Is

**Multimodal search** = finding documents by their words and by how they look: stamps, boxes, logos, handwriting.

In this lesson, the support team searches the archive of invoice pages. Most questions are about words: an invoice number, a supplier. Some are about looks: "invoices stamped PAID".

The first version only searches the text that OCR read from each page. An agent searches for "invoices stamped PAID" and gets almost nothing back. A stamp is a picture, not a word.

Press **Play**, or use **Next** to go one step at a time.

- **Frame 1:** search over OCR text. Most stamped invoices are missing.
- **Frame 2:** search over page images. They come back.

**Simple lesson:** some answers are in the words, and some are in how the page looks.

### A Real-World Example

Think about a librarian helping you find a book.

- "The one called *Kettles of the World*" → they look it up by title in the catalogue.
- "The big red one with a gold spine" → they walk to the shelf and look.

Search works the same way:

- **The catalogue** = an index of OCR text embeddings.
- **Looking at the shelf** = an index of page-image embeddings.
- **Knowing which to use** = routing each question to the right index.

**Simple lesson:** keep two indexes, and send each question to the one that holds its answer.

## What You Will Learn

By the end of this lesson, you will be able to:

- explain what OCR text misses
- build a page-image index
- measure **recall**
- route each question to an index
- re-rank the top results
- write answers that cite their pages
- search with your own questions

## Two Kinds of Embedding

An **embedding** = a list of numbers that stands for the meaning of something. Search finds the items whose numbers are closest to the question's.

| Index | What is embedded | Good for |
|---|---|---|
| OCR text | The words read from each page | Numbers, names, products |
| Page image | A picture of the whole page | Stamps, boxes, logos, handwriting, layout |

**Recall** = of all the right pages, how many the search found (in its top 5). The lab's target is at least 80% for every question.

Step through all six frames.

- **Left:** the top results from the OCR text index.
- **Right:** the top results from the page-image index.
- **Ticks:** right pages.

**Page images blur small text.** An image embedding sees the whole page at low detail. It finds "a stamp near the top", but it can confuse INV-2390 with INV-2398. Exact numbers and names belong in the text index.

**Simple lesson:** neither index wins every question.

## Route Each Question

Step through all three frames.

- **Text only:** layout questions fail.
- **Images only:** the invoice-number question fails.
- **Route each question:** both kinds pass.

A simple router looks for layout words: stamp, box, handwritten, logo, top, bottom, signature. A small classifier model can do the same job for open questions.

## Re-Rank, Then Answer With Sources

Embeddings are fast but rough. A **re-ranker** reads the question and each of the top 10 pages together, and puts them in a better order. It is slower, so it only looks at a few pages.

Then an LLM writes the answer from the top pages. Every fact must name its page, like `[p06]`, so a person can open it and check.

| Step | What it does | Cost |
|---|---|---|
| Embedding search | Finds about 10 likely pages | Very cheap |
| Re-ranker | Reorders those 10 | A little more |
| Answer with citations | Writes the answer from the top 3, naming each page | One LLM call |

**Search results are not answers.** A retrieved page still needs the checks from Lessons 3 and 5 before you use a value from it. And remember the right to delete: if a customer asks you to remove their data, remove it from both indexes.

**Easy way to remember:** words → text, looks → image, not sure → both.

Click the button and look at the **Pipeline** tab.

- **The question list:** pick one, or type your own question in the search box.
- **The results:** right pages are ticked, with each page's id.
- **The answer:** written from the top pages, with or without citations.

### Lab Exercise: Route Each Question to the Right Index

Your task: choose an index for word questions and for layout questions in the **Use** stage, then press **Check**.

Click the button and open the **Pipeline** tab.

- **Word questions:** OCR text, page image, or hybrid (both).
- **Layout questions:** the same choice.
- **Re-rank and cite:** turn each on or off.
- **To pass:** at least 80% recall for both kinds of question, and a citation for every fact in the answer.

## Build It in TypeScript

```ts
const LAYOUT = /stamp|box|handwritten|logo|top|bottom|signature|looks|layout/i;

export function routeQuery(question: string): "text" | "image" {
  return LAYOUT.test(question) ? "image" : "text";
}
```

Click the button and open the **Code** tab.

- **The starter:** sends every question to the text index.
- **Run on the media bank:** your function routes 10 questions.

## TL;DR

1. **OCR text** holds the words; the **page image** holds the looks.
2. An **embedding** turns a page or a question into numbers you can compare.
3. **Recall** = right pages found ÷ right pages.
4. Page images blur **exact numbers**; OCR text misses **stamps and layout**.
5. **Route** each question, or search both when unsure.
6. **Re-rank** the top 10, and **cite** a page for every fact.

## Common mistakes

| Mistake | Wrong idea | Correct idea | What goes wrong |
|---|---|---|---|
| OCR text only | "Everything is in the text." | Index page images too. | Stamped invoices are never found. |
| Page images only | "Images hold everything." | Keep the text index for numbers. | INV-2390 returns the wrong invoice. |
| No recall test | "The results look right." | Measure recall on known questions. | Missing pages go unnoticed. |
| An answer with no sources | "The answer reads well." | Cite the page for every fact. | Nobody can check it. |
| Using a result as an answer | "Search found it, so it is right." | Check the value as in Lesson 5. | A wrong total is used. |
| Deleting from one index | "The file is gone." | Delete from every index. | Personal data stays searchable. |

## Check Your Understanding

## Exercises

## What to Remember

- Words live in the text; looks live in the image.
- Keep two indexes.
- Route each question.
- Measure recall.
- Cite a page for every fact.
- Check results before you use them.

## Further Reading

- [ColPali: Efficient Document Retrieval with Vision Language Models](https://arxiv.org/abs/2407.01449): searching page images directly.
- [Transformers.js documentation, by Hugging Face](https://huggingface.co/docs/transformers.js/index): text and image embedding models in the browser.

Next: **Cost, Privacy and On-Device**, where you decide what may leave the customer's phone.