Why does OCR text miss "invoices stamped PAID"?

A stamp is a shape, often not read as text

Multimodal Search

By the Syncfusion DevTutorials team · Last updated October 27, 2026

Search a document archive by words and by looks: OCR text and page-image embeddings, routing, re-ranking, and answers that cite their pages.

What Multimodal Search Is

Multimodal search = finding documents by their words and by how they look: stamps, boxes, logos, handwriting.

In this lesson, the support team searches the archive of invoice pages. Most questions are about words: an invoice number, a supplier. Some are about looks: "invoices stamped PAID".

The first version only searches the text that OCR read from each page. An agent searches for "invoices stamped PAID" and gets almost nothing back. A stamp is a picture, not a word.

Press Play, or use Next to go one step at a time.

Frame 1: search over OCR text. Most stamped invoices are missing.
Frame 2: search over page images. They come back.

Simple lesson: some answers are in the words, and some are in how the page looks.

A Real-World Example

Think about a librarian helping you find a book.

"The one called Kettles of the World" → they look it up by title in the catalogue.
"The big red one with a gold spine" → they walk to the shelf and look.

Search works the same way:

The catalogue = an index of OCR text embeddings.
Looking at the shelf = an index of page-image embeddings.
Knowing which to use = routing each question to the right index.

Simple lesson: keep two indexes, and send each question to the one that holds its answer.

What You Will Learn

By the end of this lesson, you will be able to:

explain what OCR text misses
build a page-image index
measure recall
route each question to an index
re-rank the top results
write answers that cite their pages
search with your own questions

Two Kinds of Embedding

An embedding = a list of numbers that stands for the meaning of something. Search finds the items whose numbers are closest to the question's.

Index	What is embedded	Good for
OCR text	The words read from each page	Numbers, names, products
Page image	A picture of the whole page	Stamps, boxes, logos, handwriting, layout

Recall = of all the right pages, how many the search found (in its top 5). The lab's target is at least 80% for every question.

Step through all six frames.

Left: the top results from the OCR text index.
Right: the top results from the page-image index.
Ticks: right pages.

Page images blur small text. An image embedding sees the whole page at low detail. It finds "a stamp near the top", but it can confuse INV-2390 with INV-2398. Exact numbers and names belong in the text index.

Simple lesson: neither index wins every question.

Route Each Question

Step through all three frames.

Text only: layout questions fail.
Images only: the invoice-number question fails.
Route each question: both kinds pass.

A simple router looks for layout words: stamp, box, handwritten, logo, top, bottom, signature. A small classifier model can do the same job for open questions.

Re-Rank, Then Answer With Sources

Embeddings are fast but rough. A re-ranker reads the question and each of the top 10 pages together, and puts them in a better order. It is slower, so it only looks at a few pages.

Then an LLM writes the answer from the top pages. Every fact must name its page, like [p06], so a person can open it and check.

Step	What it does	Cost
Embedding search	Finds about 10 likely pages	Very cheap
Re-ranker	Reorders those 10	A little more
Answer with citations	Writes the answer from the top 3, naming each page	One LLM call

Search results are not answers. A retrieved page still needs the checks from Lessons 3 and 5 before you use a value from it. And remember the right to delete: if a customer asks you to remove their data, remove it from both indexes.

Easy way to remember: words → text, looks → image, not sure → both.

Click the button and look at the Pipeline tab.

The question list: pick one, or type your own question in the search box.
The results: right pages are ticked, with each page's id.
The answer: written from the top pages, with or without citations.

Lab Exercise: Route Each Question to the Right Index

Your task: choose an index for word questions and for layout questions in the Use stage, then press Check.

Click the button and open the Pipeline tab.

Word questions: OCR text, page image, or hybrid (both).
Layout questions: the same choice.
Re-rank and cite: turn each on or off.
To pass: at least 80% recall for both kinds of question, and a citation for every fact in the answer.

Build It in TypeScript

const LAYOUT = /stamp|box|handwritten|logo|top|bottom|signature|looks|layout/i;

export function routeQuery(question: string): "text" | "image" {
  return LAYOUT.test(question) ? "image" : "text";
}

Click the button and open the Code tab.

The starter: sends every question to the text index.
Run on the media bank: your function routes 10 questions.

TL;DR

OCR text holds the words; the page image holds the looks.
An embedding turns a page or a question into numbers you can compare.
Recall = right pages found ÷ right pages.
Page images blur exact numbers; OCR text misses stamps and layout.
Route each question, or search both when unsure.
Re-rank the top 10, and cite a page for every fact.

Common mistakes

Mistake	Wrong idea	Correct idea	What goes wrong
OCR text only	"Everything is in the text."	Index page images too.	Stamped invoices are never found.
Page images only	"Images hold everything."	Keep the text index for numbers.	INV-2390 returns the wrong invoice.
No recall test	"The results look right."	Measure recall on known questions.	Missing pages go unnoticed.
An answer with no sources	"The answer reads well."	Cite the page for every fact.	Nobody can check it.
Using a result as an answer	"Search found it, so it is right."	Check the value as in Lesson 5.	A wrong total is used.
Deleting from one index	"The file is gone."	Delete from every index.	Personal data stays searchable.

Check Your Understanding

Exercises

What to Remember

Words live in the text; looks live in the image.
Keep two indexes.
Route each question.
Measure recall.
Cite a page for every fact.
Check results before you use them.