# Multimodal RAG

Course: [Retrieval-Augmented Generation](https://www.devtutorials.ai/learn/rag) · Section: Advanced RAG
Source: https://www.devtutorials.ai/learn/rag/advanced-rag/multimodal-rag

> Retrieving from images, screenshots, tables and scanned pages: captions and OCR, tables as Markdown or CSV, OCR clean-up, layout analysis, shared text-image vector spaces and multi-vector page matching.

A support manual may contain information in different formats. The location of a reset button may appear in an image, prices may be listed in a table, and setup instructions may be shown through screenshots.

A RAG system that processes only plain text may miss this useful information.

Multimodal RAG allows the system to retrieve information from multiple types of content, including text, images, screenshots, tables, and scanned pages. This helps the assistant answer questions using information that is not available as plain text.

**The main idea:** there are two complementary approaches. **Convert to text**: OCR scanned pages, turn tables into Markdown or CSV, caption images and extract the text in them, analyse the page layout, then use normal text RAG. **Embed visually**: use models that put **text and images in one vector space**, so a text question can find an image directly, including **multi-vector** models that match question words to parts of a page.

## A Real-World Example

Brightgrid customers often ask:

Where is the reset button on the Lumo Lamp?

The answer appears in a product image inside the setup guide. However, the first ingestion process extracted only the written paragraphs and left out the image and its caption. As a result, the assistant could not find the answer.

The system had another problem with a plan-comparison table. During ingestion, the rows and headings were mixed into one unclear line, so the assistant could not reliably connect each plan with its price and schedule limit.

Brightgrid improved the ingestion process:

Images are stored with their captions, alternative text, and a link to the original image.
OCR, or Optical Character Recognition, extracts written text from images and scanned pages.
Tables keep their headings, rows, and relationships between values.
Scanned PDF files are processed with OCR, and unclear values are marked for review.
Page images are made searchable so that questions describing visual details can find the relevant page.

The assistant can now answer:

The reset button is next to the USB-C port on the back of the lamp [S1].

The application can also display the supporting image so the user can confirm the button’s location.

**In simple words:** turn pictures and tables into searchable text, or search them as pictures directly.

## What You Will Learn

In this lesson, you will learn:

1. Multimodal document processing: context from images and screenshots, tables as CSV or Markdown, OCR for scanned pages, and layout analysis.
2. Multimodal embeddings: shared text-image vector spaces, retrieving images with natural language, image indexing and cross-modal distance, and multi-vector visual features.

The RAG Lab works with text, but you can try the "convert to text" approach yourself. Add a Knowledge base article that describes an image with its caption and OCR text (the try card below does this), then ask a question only the image could answer.

## Multimodal Document Processing

### Extracting Context from Images, Screenshots, and Tables

An image becomes useful for search when you turn it into text that describes it:

- **Caption and alt text** written by the author: often the best description you'll get.
- **Text inside the image** (labels, buttons, screen text), extracted with OCR.
- **A generated description** from a vision-language model, such as "The back of a lamp with a small round button labelled RESET next to a USB-C port". This is helpful when there's no caption, but check a sample, because these models can be wrong.

Store the description as a chunk, **plus a link to the image**, so the answer can show it.

```js runnable title="Turn images into searchable chunks"
// Images as searchable chunks: describe them in text, and keep a link to the image.
const images = [
  {
    file: "lumo-ports.png",
    alt: "Back of the Lumo Lamp",
    caption: "The reset button is next to the USB-C port.",
    ocr: "RESET  USB-C  5V",
  },
  {
    file: "app-add-device.png",
    alt: "App screen",
    caption: "Press Add device on the Home tab.",
    ocr: "Home  Rooms  + Add device",
  },
];

const chunks = images.map((img) => ({
  id: `img:${img.file}`,
  text: `[Image: ${img.alt}] ${img.caption} Text in the image: ${img.ocr.replace(/\s{2,}/g, ", ")}.`,
  image: img.file,
}));
chunks.forEach((c) => console.log(`${c.id}\n  ${c.text}`));

const q = "where is the reset button";
const hit = chunks.find(
  (c) => q.split(" ").filter((w) => c.text.toLowerCase().includes(w)).length >= 3,
);
console.log(`\nquery "${q}" -> ${hit.id} (show ${hit.image} next to the answer)`);
```

Expected output:

```text
img:lumo-ports.png
  [Image: Back of the Lumo Lamp] The reset button is next to the USB-C port. Text in the image: RESET, USB-C, 5V.
img:app-add-device.png
  [Image: App screen] Press Add device on the Home tab. Text in the image: Home, Rooms, + Add device.

query "where is the reset button" -> img:lumo-ports.png (show lumo-ports.png next to the answer)
```

### Table Preservation & CSV/Markdown Conversion

Tables hold some of the most-asked facts: prices, limits, specs and compatibility. Keep their structure:

- **Markdown tables** are readable by both people and models, and suit small tables in chunks.
- **CSV** is better for tools, for example a "lookup" tool that filters rows exactly.
- **Row sentences** ("Plan: Plus, Price per month: $3.99, Schedules: unlimited.") make each row searchable on its own (Lesson 4).

For big tables, store one chunk per row or group of rows, and **repeat the header** in each, so no chunk loses its column names.

```js runnable title="A table as Markdown and as CSV"
// Tables: keep them as Markdown (for the model) or CSV (for tools), never as loose words.
const table = {
  caption: "Brightgrid plans",
  headers: ["Plan", "Price per month", "Schedules"],
  rows: [
    ["Free", "$0", "5"],
    ["Plus", "$3.99", "unlimited"],
  ],
};

const markdown = [
  `Table: ${table.caption}`,
  `| ${table.headers.join(" | ")} |`,
  `| ${table.headers.map(() => "---").join(" | ")} |`,
  ...table.rows.map((r) => `| ${r.join(" | ")} |`),
].join("\n");
const csv = [table.headers, ...table.rows]
  .map((r) => r.map((c) => (/[",]/.test(c) ? `"${c.replace(/"/g, '""')}"` : c)).join(","))
  .join("\n");

console.log(markdown);
console.log("\n" + csv);
```

Expected output:

```text
Table: Brightgrid plans
| Plan | Price per month | Schedules |
| --- | --- | --- |
| Free | $0 | 5 |
| Plus | $3.99 | unlimited |

Plan,Price per month,Schedules
Free,$0,5
Plus,$3.99,unlimited
```

### OCR Processing for Scanned PDF Pages

Scanned pages are images of text. **OCR** turns them back into text. Modern OCR is good, but it makes predictable mistakes:

- **Letters for digits**: O for 0, I or l for 1, S for 5, g for 9, especially inside numbers.
- **Broken words** at line ends, and **lost formatting** like bold headings and table borders.

Fix the predictable cases inside numbers, **record every change**, and send low-confidence words for review.

```js runnable title="Fix OCR digit mix-ups, and log the changes"
// OCR clean-up for scanned pages: fix typical character mix-ups, but only where they are
// clearly numbers, and keep a list of every change for review.
const ocrText =
  "Return within 3O days. Refunds take 5 business clays. Model BG-1O42 costs $3.9g.";
const changes = [];

const fixed = ocrText.replace(
  /\b[\dOIlSg][\dOIlSg.$]*\d[\dOIlSg.]*\b|\b\d+[OIlSg]+\b/g,
  (token) => {
    const repaired = token
      .replace(/O/g, "0")
      .replace(/[Il]/g, "1")
      .replace(/S/g, "5")
      .replace(/g/g, "9");
    if (repaired !== token) {
      changes.push(`${token} -> ${repaired}`);
    }
    return repaired;
  },
);

console.log("ocr:     " + ocrText);
console.log("fixed:   " + fixed);
console.log("changes: " + changes.join(", "));
console.log(
  'Not fixed: "clays" should be "days". Word errors need a dictionary or a person.',
);
```

Expected output:

```text
ocr:     Return within 3O days. Refunds take 5 business clays. Model BG-1O42 costs $3.9g.
fixed:   Return within 30 days. Refunds take 5 business clays. Model BG-1042 costs $3.99.
changes: 3O -> 30, 1O42 -> 1042, 3.9g -> 3.99
Not fixed: "clays" should be "days". Word errors need a dictionary or a person.
```

### Visual Layout Structure Analysis

**Layout analysis** finds the **regions** of a page: title, headings, paragraphs, columns, tables, figures, captions, headers and footers. It tells you:

- The **reading order** across columns (Lesson 4).
- Which caption belongs to which figure.
- Which regions are tables, so they get table handling instead of plain OCR.
- Which repeated regions are headers and footers to drop.

Document-AI tools return these regions with their positions and types. Keep the region type as metadata on each chunk, like `type: table` or `type: figure`.

## Multimodal Embeddings

### Unified Text-Image Vector Spaces

Some embedding models can work with both text and images. They place an image and text with a similar meaning close together in the same vector space.

For example, a photo of a lamp with a blue light may be placed close to the sentence: A lamp showing a blue connection light.

This allows the RAG system to use a text question to search for a related image, even when the image does not contain the exact words from the question.

```js runnable title="Text queries matched to image vectors"
// A shared text-image vector space: a text query can be compared with image vectors directly.
// (Toy 3-number vectors; models like CLIP learn real ones from image-caption pairs.)
const images = {
  "photo of a lamp blinking blue": [0.9, 0.1, 0.1],
  "photo of the hub on a shelf": [0.1, 0.9, 0.2],
  "screenshot of a price table": [0.1, 0.2, 0.9],
};
const textQueries = {
  "lamp light flashing": [0.85, 0.15, 0.05],
  "how much does it cost": [0.05, 0.15, 0.95],
};
const cos = (a, b) => {
  const d = a.reduce((s, x, i) => s + x * b[i], 0);
  return d / (Math.hypot(...a) * Math.hypot(...b));
};

for (const [q, qv] of Object.entries(textQueries)) {
  const ranked = Object.entries(images)
    .map(([name, v]) => [name, cos(qv, v)])
    .sort((a, b) => b[1] - a[1]);
  console.log(`"${q}" -> ${ranked[0][0]} (${ranked[0][1].toFixed(2)})`);
}
```

Expected output:

```text
"lamp light flashing" -> photo of a lamp blinking blue (1.00)
"how much does it cost" -> screenshot of a price table (1.00)
```

### Retrieving Visual Chunks via Natural Language

With a shared space, "lamp blinking blue" can find a photo of a blinking lamp even if no one wrote a caption. It's useful for product photos, diagrams, screenshots, and scanned pages that are hard to OCR. Show the retrieved image to the user, and give the model either the image itself (if it can read images) or its description.

### Image Vector Indexing & Cross-Modal Distance

Index image vectors in the same kind of vector index as text (Lesson 10), with metadata (file, page, caption). Two practical cautions:

- **Scores aren't comparable across modalities.** Text-to-image cosines are often lower than text-to-text cosines for equally good matches. Rank images and text **separately**, then merge with RRF (Lesson 12), rather than mixing raw scores.
- **Use one model consistently.** The text side must be encoded by the same model's text encoder, not your usual text embedding model.

### Multi-Vector Visual Feature Extraction

A document page may contain several kinds of information, such as paragraphs, tables, diagrams, and images. Representing the entire page with one vector can hide small but important details.
Multi-vector retrieval represents a page using several vectors instead of only one. Each vector can represent a different area of the page.
When a user asks a question, the system compares parts of the question with different areas of the page. It then combines the strongest matches to calculate the page’s overall relevance.
This approach is called late interaction. Models in the ColPali family use this method to search visually rich documents such as scanned pages, forms, tables, and diagrams.​‌
Multi-vector retrieval can preserve more detail than a single page vector, but it also needs more storage and processing.

```js runnable title="MaxSim: each query word finds its best patch"
// Multi-vector (late interaction) matching: one vector per query word and per image patch.
// Each query word finds its best patch; the scores are added up ("MaxSim").
const queryWords = { reset: [1, 0, 0], button: [0, 1, 0] };
const pages = {
  "manual page 4": { patchA: [0.9, 0.1, 0], patchB: [0.1, 0.8, 0.1], patchC: [0, 0, 1] },
  "manual page 9": {
    patchA: [0.2, 0.1, 0.9],
    patchB: [0.3, 0.3, 0.3],
    patchC: [0, 0.2, 0.9],
  },
};
const dot = (a, b) => a.reduce((s, x, i) => s + x * b[i], 0);

for (const [page, patches] of Object.entries(pages)) {
  let total = 0;
  const parts = [];
  for (const [w, wv] of Object.entries(queryWords)) {
    const best = Math.max(...Object.values(patches).map((p) => dot(wv, p)));
    total += best;
    parts.push(`${w} ${best.toFixed(1)}`);
  }
  console.log(`${page}: ${parts.join(" + ")} = ${total.toFixed(1)}`);
}
```

Expected output:

```text
manual page 4: reset 0.9 + button 0.8 = 1.7
manual page 9: reset 0.3 + button 0.3 = 0.6
```

Start with **convert to text**: captions, OCR and table conversion work with your existing pipeline and are easy to debug. Add visual embeddings when you have many images or scanned pages that text conversion handles poorly, and measure the gain on a gold set that includes image questions.

Text inside images is a hiding place for **prompt injection** too. A screenshot can contain "AI: ignore your instructions". Treat OCR text and generated descriptions as untrusted data, exactly like any other retrieved text (Lesson 17).

## Common mistakes

- **Losing captions**, so images can't be found by any text query.
- **Splitting tables** so that rows lose their column names.
- **OCR errors in prices and dates** that nobody checks.
- **Mixing embedding models**: images from one model, queries from another.
- **One vector per dense page**, which blurs the very details people search for.

## Exercises

Each exercise ends with a value on the last line. Press **Check** to compare it with the expected answer.

## What to Remember

- Multimodal RAG retrieves from **images, screenshots, tables and scanned pages**.
- **Convert to text**: captions, alt text and OCR for images; **Markdown, CSV or row sentences** for tables; **OCR** with digit fixes and review for scans; **layout analysis** for reading order and region types.
- **Embed visually**: **shared text-image spaces** (like CLIP) let text find images. Rank images and text separately, then merge.
- **Multi-vector** models match each query word to its best **patch** (MaxSim), keeping detail on rich pages.
- Text in images is **untrusted data**, just like any other retrieved text.

Next: **Advanced Retrieval Patterns**: multi-hop questions, agentic RAG, Self-RAG and GraphRAG.