# Document and Media Context

Course: [Context Engineering](https://www.devtutorials.ai/learn/context-engineering) · Section: Memory, Media and Tools
Source: https://www.devtutorials.ai/learn/context-engineering/memory-media-tools/document-media-context

> How PDFs, Word files, spreadsheets, slides, images, audio and video become labelled text a model can use, how to keep unclear parts marked as unclear, and a four-step framework for any file.

You have a big PDF file with many pages. How do you turn it into something the AI model can use? You can't just say "here's a PDF" to the model.

AI models only read text. So images, videos, audio files, and documents must be turned into text or labels the model understands.

**The main idea:** pull out the words and images, label where they came from, mark how sure we are, and tell the model what to trust.

## A Real-World Example

Think about a restaurant manager.

- A customer sends a picture of a broken plate.
- The manager doesn't just look at the picture. She says: "I see this is the blue plate from the Premium line, broken at the rim. This is our problem, I'm 80% sure."
- She tags it: broken, high priority, our fault.
- Next week, she counts broken plates by tag, to see if there's a pattern.

The tags, the labels, and the confidence level are as important as the picture.

**In simple words:** pull out the data, mark where it came from, say how sure you are, and keep a record.

## What You Will Learn

In this lesson, you will learn:

1. How to turn PDFs, Word files, spreadsheets, and presentations into plain text.
2. How to pull text and images out of pictures.
3. How to turn audio and video into text.
4. How to mark where data came from, and how sure it is.
5. The four steps: pull it out, label it, sum it up, check it.

## Turning Documents Into Text

### PDFs

A PDF is a picture of a page. To turn it into text that the model can read, you must:

1. Get every page.
2. Read the text or use image reading (OCR) to get the words.
3. Label which page it came from.
4. Mark if it has images or tables that need special work.

```js runnable title="Label each page with its source"
const pages = [
  { number: 1, text: "Welcome to Brightgrid.", hasImage: false },
  { number: 2, text: "Pricing: Free plan includes 5,000 rows.", hasImage: false },
  { number: 3, text: "[Diagram of features]", hasImage: true },
];

for (const p of pages) {
  console.log(`Page ${p.number}: ${p.hasImage ? "[has image] " : ""}${p.text.substring(0, 40)}...`);
}
```

Expected output:

```text
Page 1: Welcome to Brightgrid.
Page 2: Pricing: Free plan includes 5,000 rows.
Page 3: [has image] [Diagram of features]...
```

Every page gets a label. The model later knows: "this came from page 2 of the PDF, not from the internet."

### Word Documents and Spreadsheets

Word files are similar to PDFs. Spreadsheets are different: they have rows and columns.

For a spreadsheet:
- Pick the rows and columns you need.
- Turn them into a table or list.
- Say which sheet and row.

```js runnable title="Turn a spreadsheet into a table the model sees"
const sheet = [
  ["Customer", "Plan", "Monthly cost"],
  ["Alice", "Free", "$0"],
  ["Bob", "Pro", "$20"],
  ["Carol", "Team", "$50"],
];

for (const row of sheet) {
  console.log(row.join(" | "));
}
```

Expected output:

```text
Customer | Plan | Monthly cost
Alice | Free | $0
Bob | Pro | $20
Carol | Team | $50
```

The model sees a clear table with labels. It knows the second row is Alice, and she is on the Free plan.

### Presentations

A PowerPoint or Keynote is a set of slides. For each slide:

1. Get the text on the slide.
2. Note if there are charts, graphs, or images.
3. Say what slide number.

## Reading Text and Images From Pictures

### OCR: Reading Text From Images

**OCR** means "read text from a picture". A tool looks at an image and finds the words.

For example, a customer sends a picture of a receipt from a store. OCR pulls out the words: "Total: $45.99, Date: 2026-01-15."

```js runnable title="Mark OCR confidence"
const images = [
  { id: "img-1", text: "Total: $45.99", confidence: 0.95 },
  { id: "img-2", text: "Paid by Visa", confidence: 0.78 },
  { id: "img-3", text: "[blurry number]", confidence: 0.42 },
];

for (const i of images) {
  console.log(`${i.id}: "${i.text}" (${Math.round(i.confidence * 100)}% sure)`);
}
```

Expected output:

```text
img-1: "Total: $45.99" (95% sure)
img-2: "Paid by Visa" (78% sure)
img-3: "[blurry number]" (42% sure)
```

When OCR is unsure (below 70%), mark it. The model then knows: "this number might be wrong."

### Diagrams and Charts

When a picture shows a diagram or chart:

1. Try OCR to pull text.
2. If it's a chart, describe what it shows: "bar chart of sales by month, January highest."
3. If OCR fails, describe the picture by eye.

### Handwriting and Receipts

Handwriting is hard for OCR. If a receipt has handwritten notes, mark them as "hard to read" and give the AI what it might say.

## Turning Audio and Video Into Text

### Audio Files

An audio file (MP3, WAV) must be turned into words. A **transcription** tool listens to the audio and writes down what it hears.

```js runnable title="Mark a transcript with speaker and time"
const transcript = [
  { time: "0:00", speaker: "Agent", text: "How can I help you?" },
  { time: "0:05", speaker: "Customer", text: "I want to cancel my plan." },
  { time: "0:12", speaker: "Agent", text: "I can help with that. May I ask why?" },
];

for (const t of transcript) {
  console.log(`[${t.time}] ${t.speaker}: ${t.text}`);
}
```

Expected output:

```text
[0:00] Agent: How can I help you?
[0:05] Customer: I want to cancel my plan.
[0:12] Agent: May I ask why?
```

Now the model can read the conversation. It knows who spoke and when.

### Video Files

A video is pictures plus audio. To turn it into text:

1. Get the audio and turn it into a transcript (like above).
2. Get the pictures that show important moments (key frames).
3. Describe what happens in each key frame.
4. Tie the description to the time in the video.

```js runnable title="Build a video summary with time stamps"
const keyFrames = [
  { time: "0:00", description: "Customer on screen, pointing at screen" },
  { time: "0:08", description: "Close-up of error message: 'Quota exceeded'" },
  { time: "0:15", description: "Agent appears on screen, nods" },
];

for (const k of keyFrames) {
  console.log(`[${k.time}] ${k.description}`);
}
```

Expected output:

```text
[0:00] Customer on screen, pointing at screen
[0:08] Close-up of error message: 'Quota exceeded'
[0:15] Agent appears on screen, nods
```

The model now knows: at 8 seconds, an error message appeared.

## Labeling Source and Confidence

### Trust Levels

Mark every piece of data with a trust level:
- **System**: you made it, you trust it fully.
- **Official**: from your company's own sources.
- **Community**: from users or forums, be careful.
- **Untrusted**: from outside, could be wrong.

### Confidence Scores

When a tool reads data (like OCR), it says how sure it is, from 0 to 1:
- **0.9 to 1.0**: very sure
- **0.7 to 0.9**: pretty sure
- **0.5 to 0.7**: not very sure
- **Below 0.5**: don't trust it

```js runnable title="Pull data with trust and confidence"
const data = [
  { source: "official", confidence: 0.99, text: "Free plan: 5,000 rows" },
  { source: "ocr_receipt", confidence: 0.85, text: "Total: $99" },
  { source: "ocr_handwriting", confidence: 0.42, text: "Paid by ???" },
];

for (const d of data) {
  const sure = d.confidence > 0.7 ? "trust this" : "CHECK THIS";
  console.log(`${d.source}: "${d.text}" (${sure})`);
}
```

Expected output:

```text
official: "Free plan: 5,000 rows" (trust this)
ocr_receipt: "Total: $99" (trust this)
ocr_handwriting: "Paid by ???" (CHECK THIS)
```

### Provenance Labels

**Provenance** means "where it came from". Always label:
- The file name or page.
- The time, if important.
- What tool pulled it out (OCR, transcription, etc).

## The Four-Step Process

### Step 1: Extract

Pull the data out of the file. For a PDF, get each page. For an image, run OCR or describe it. For audio, run transcription. For video, get the key moments.

### Step 2: Label

Tag every piece with:
- What file or page it came from.
- Its trust level.
- Its confidence score (if a tool pulled it).
- When it was created or updated.

### Step 3: Summarize

For long documents, pull out the key points, not the whole thing. A 50-page handbook should become 10 key sections the question needs.

### Step 4: Verify

Double-check:
- Did OCR read it right, or did it guess?
- Is it up to date?
- Is the label clear?
- Is the data good enough to use?

```js runnable title="The four steps in order"
const steps = [
  { step: "Extract", action: "Get pages, run OCR, transcribe audio, find key moments" },
  { step: "Label", action: "Mark file, source, confidence, date, trust" },
  { step: "Summarize", action: "Pull key points for this question" },
  { step: "Verify", action: "Check OCR, dates, labels, quality" },
];

for (const s of steps) {
  console.log(`${s.step}: ${s.action}`);
}
```

Expected output:

```text
Extract: Get pages, run OCR, transcribe audio, find key moments
Label: Mark file, source, confidence, date, trust
Summarize: Pull key points for this question
Verify: Check OCR, dates, labels, quality
```

**Easy Way to Remember:** for every file, ask **what does it say, where did it come from, and how sure are we?**

## Multimodal Risks

### OCR Mistakes

OCR reads pictures, but it can be wrong:
- A "0" might be read as "O".
- Handwriting is very hard.
- Small text can fail.
- Rotated text can be missed.

Always mark low confidence. When confidence is below 70%, tell the model: "this might be wrong."

### Mixing Old and New

If you pull data from many files or pages:
- Mark the date of each one.
- If they don't match, tell the model they differ.
- Don't hide old data.

### Format Confusion

If a picture shows a table, OCR might:
- Read it as plain text.
- Miss the rows and columns.
- Get the order wrong.

Run OCR, but also describe what kind of table it is, so the model knows.

### Video vs. Audio

If you have a video:
- Get the audio transcript.
- Also describe what is on the screen.
- Link them by time.

Don't assume the audio alone tells the whole story. The customer might be showing something on the screen while they talk.

Text read from images and audio is a best guess, not a fact. Keep its confidence and source, and check important values before acting on them.

## Common mistakes

| Mistake | Wrong idea | Correct idea | What goes wrong |
|---|---|---|---|
| Raw file | "Send the PDF, the model will read it." | Pull text and images, label them. | The model can't read binary files. |
| No labels | "Where it came from doesn't matter." | Mark file, page, date, source, trust. | The model mixes up old and new data. |
| No confidence | "If OCR pulls it, it must be right." | Mark confidence, hide low scores. | A guessed number sounds as sure as a real one. |
| Hidden dates | "The date on a page is obvious." | Mark the date and version of each source. | Old pricing rules the answer. |
| Whole documents | "Send all pages, let the model choose." | Pull the sections this question needs. | Key facts are buried in pages of noise. |
| No verification | "OCR is good enough." | Check important numbers by hand. | Wrong numbers enter the knowledge base. |

## Exercises

Each exercise ends with a value on the last line. Press **Check** to compare it with the expected answer.

## What to Remember

- **Extract**: turn PDFs, images, audio, and video into text and labels.
- **Label** every piece: file, page, date, source, trust level, confidence score.
- **Summarize** long documents: pull key points, not everything.
- **Verify** data, especially OCR and transcription. Mark low confidence.
- Mark when data is **old or mixed**. The model then knows.

Next: **Memory Management**, how to keep facts from a long chat, and let them expire when they're no longer needed.