What Multimodal Models Can and Can't Do
By the Syncfusion DevTutorials team · Last updated
A model that can see is not a model that is right. Run ten claim items through a multimodal LLM and through specialist models, compare accuracy, cost and time, and learn four questions that pick the right tool.
What Multimodal AI Is
Multimodal AI = AI that takes in more than text: photos, scanned pages, screenshots, charts, audio and video.
In this lesson, you will see what these models do well and where they fail. You will also learn how to pick the right tool for each job.
A customer claims a refund for order #A-1042. They upload a photo of a creased receipt. The app asks a multimodal model for the total.
The model answers 1,860. The right answer is 186.00. It sounds just as sure as it does about the shop name and the date.
Press Play, or use Next to go one step at a time.
- Frame 1: the receipt, with a very small total.
- Frame 2: what the model returned, and how sure it was.
- Frame 3: why it happened: the decimal point is only a few pixels wide.
- Frame 4: a specialist model reads the same receipt correctly, for less money.
Simple lesson: a model that can see is not a model that is right.
A Real-World Example
Think about a hospital.
- A family doctor can look at almost any problem. They listen, look and think about the whole person.
- A blood test machine does one thing. It measures exact numbers, fast and cheaply, every time.
You would not ask the doctor to count your blood cells by eye. You would not ask the machine what is wrong with you.
AI works the same way:
- The family doctor = a multimodal LLM. It handles anything, roughly. It is good at meaning and judgment.
- The test machines = specialist models: OCR for printed text, speech-to-text for audio, a detector for finding and counting things.
Good teams use both. The doctor reads the test results; the machine never guesses a diagnosis.
Simple lesson: use the generalist for judgment, and specialists for exact numbers.
What You Will Learn
By the end of this lesson, you will be able to:
- explain why a multimodal model can be wrong and still sound sure
- say how images, pages and audio become tokens, and what that costs
- compare a multimodal LLM with three specialist models on ten real claim items
- pick the right tool for a job with four questions
- name the six stages of a multimodal pipeline, and the four you write yourself
The lab uses the course's claims-and-receipts app. Customers send photos of receipts, PDF invoices, damage photos, screenshots of errors and voice notes. Your job is to turn them into correct data.
Everything Becomes Tokens
A model cannot "look" at a photo the way you do. The photo is cut into small pieces, and each piece becomes tokens: the same units the model uses for text.
Step through all four frames.
- The bar: the tokens one receipt costs.
- Frame 1: the full photo.
- Frames 2 and 3: a smaller photo, then a crop of just the total.
- Frame 4: the text from an OCR model instead of the photo.
That one fact gives the three trade-offs you will meet in every lesson:
| Trade-off | What it means | Lesson |
|---|---|---|
| Resolution vs cost | More pixels or frames = more tokens, more money, more waiting | 2 |
| Context vs focus | Sending the whole document is easy; a crop is often more accurate | 2, 3 |
| Generalist vs specialist | An LLM handles anything roughly; a specialist does one job precisely and cheaply | This lesson |
The prices in the lab are illustrative. They follow the shape of real pricing (per token, per page, per minute), but they are not any one vendor's. Real prices change often, so the course keeps them in one file that is checked before every release.
Simple lesson: every pixel, page and second of audio is paid for in tokens.
Generalist or Specialist
Here is what each tool is for. You will use all four in this course.
| Tool | Good at | Weak at | Returns |
|---|---|---|---|
| Multimodal LLM | Meaning, judgment, handwriting, context | Small print, exact counts, long tables | Words |
| OCR + layout model | Printed text, numbers, tables, positions | Meaning, handwriting | Text with boxes |
| Speech-to-text model | Turning speech into words, fast | Product names it has never heard | Words with timestamps |
| Object detector | Finding and counting things | Saying what is wrong with them | Boxes with labels |
Now run ten claim items through the LLM and through the right specialist.
Step through all ten frames.
- Left: what the multimodal LLM read, and how sure it was.
- Right: what the specialist read.
- The ticks: which values match the right answer.
- On the right of the board: fields right, cost and time for each tool.
The LLM wins five items: an error screenshot, a chart, a damage photo, a handwritten tip and a voice note with a product name. The specialists win the other five: two receipts, a long invoice, a count and a plain voice note.
Click the button and look at the Media tab.
- The switch above the receipt: change the tool.
- The boxes: OCR shows where each value came from. The LLM shows no boxes: it only answers in words.
- The table: each value, the right answer, and how sure the tool was.
Simple lesson: neither tool wins every job. Each one fails in its own way.
Silent Errors
Some mistakes are worse than others.
- A flagged error is a wrong value the tool was unsure about. You can send it to a person to check.
- A silent error is a wrong value the tool was sure about. Nothing flags it, so it goes straight into the claim.
The 1,860 total was a silent error: 91% sure, and ten times too much money.
Open the Accuracy tab and find the silent errors column.
- Always the LLM: several silent errors.
- Best tool per item: none, and about a third of the cost.
Never trust a value because the model sounds sure. Every value you extract needs a check, a confidence signal and a path to human review. Lessons 5 and 6 build all three.
Simple lesson: a confident mistake is the most expensive kind.
How to Pick the Tool
You do not need to try every tool on every item. Ask four questions, in order. Stop at the first yes.
Step through all five frames.
- The row of questions: grey = not asked yet, green = the first yes.
- The answer: the tool that question points to.
| # | Question | If yes, use |
|---|---|---|
| 1 | Does it need meaning, judgment or handwriting? | Multimodal LLM |
| 2 | Is it speech to words? | Speech-to-text |
| 3 | Is it finding or counting things? | Object detector |
| 4 | Is it exact printed text, numbers or tables? | OCR + layout |
Ask question 1 first. A damaged-parcel photo is about objects, but the job is a judgment: "is it damaged?" A detector would only say "box".
Easy way to remember: meaning first, then speech, then things, then text.
Lab Exercise: Right Tool for Ten Items
Your task: pick a tool for each of the ten claim items. Then press Check.
Click the button and open the Right tool tab.
- Each row: choose a tool. The row shows fields right and cost as you pick.
- The two lines under the table: your picks against always using the LLM.
- To pass: the best tool for at least 8 of 10 items, cheaper than always-LLM, and at least as many fields right.
Simple lesson: pick the tool from the job, not from what is newest.
The Pipeline Is the Product
Real multimodal features are not one call to one model. They are a pipeline of six stages.
Step through all six frames.
- Each stage: what it does to the creased receipt.
- Frame 4: the check that would have caught 1,860.
| Stage | What it does | Who does it |
|---|---|---|
| Ingest | Decode, straighten, crop, split pages, cut audio | You |
| Perceive | Run a specialist, an LLM, or both | A model |
| Structure | Check the answer fits a schema | You |
| Verify | Cross-checks, confidence, where it came from | You |
| Route | Accept, or send to a person | You |
| Use | Store, search, act or speak | The app |
You write four of the six stages. Those are where most accuracy is won or lost.
Simple lesson: the model is one stage. The pipeline around it is the product.
Build It in TypeScript
The pick rule is a small function. Here it is as TypeScript. The third exercise below asks you to write it yourself.
type Tool = "llm" | "ocr" | "asr" | "det";
interface Job {
needsMeaning: boolean; // judgment, context
handwriting?: boolean;
speech: boolean;
objects: boolean; // find or count things
exactText: boolean; // printed text, numbers, tables
}
function pickTool(job: Job): Tool {
if (job.needsMeaning || job.handwriting) return "llm"; // 1. meaning first
if (job.speech) return "asr"; // 2. then speech
if (job.objects) return "det"; // 3. then things
if (job.exactText) return "ocr"; // 4. then text
return "llm";
}Specialists can also run on the user's own device, in the browser. This is how the lab's Run on my device button reads a receipt line with an OCR model, using Transformers.js:
import { pipeline } from "@huggingface/transformers";
// Downloads the model the first time, then uses the browser's copy.
const ocr = await pipeline("image-to-text", "Xenova/trocr-small-printed", {
device: "webgpu", // or "wasm" when the machine has no WebGPU
dtype: "fp16", // "q8" on the CPU: smaller and faster there
});
const [result] = await ocr(totalLineImage); // a crop of the receipt's total line
console.log(result.generated_text); // "TOTAL 186.00"The lab runs this in a Web Worker, so the page does not freeze while the model works.
Open Settings to see what your machine can run.
- Your machine: your browser, GPU, memory and free storage.
- Runs on: WebGPU (your GPU) when it is available, otherwise WebAssembly (your CPU).
- Each job: the recommended model for your machine, with its download size and memory.
- Download: nothing downloads until you press it.
On-device models are optional. Graded work in this course uses recorded outputs, so you get the same score on any laptop, with or without a GPU.
Simple lesson: route each job to the right tool in code, and check what your machine can run before you download a model.
TL;DR
- Multimodal AI = AI that takes in photos, pages, screenshots, audio and video.
- Everything becomes tokens = more pixels, pages or seconds cost more money and time.
- Generalist = a multimodal LLM: meaning, judgment, handwriting.
- Specialists = OCR for printed text, speech-to-text for audio, a detector for finding and counting.
- Silent error = a wrong value the tool was sure about. The most dangerous kind.
- The four questions = meaning? speech? things? text? Stop at the first yes.
- The pipeline = ingest, perceive, structure, verify, route, use. You write four of them.
Common mistakes
| Mistake | Wrong idea | Correct idea | What goes wrong |
|---|---|---|---|
| Sending everything to the LLM | "It can see, so it can read anything." | Use a specialist for exact text, counts and plain speech. | Silent errors in totals and counts, at almost three times the cost. |
| Trusting confidence | "It said 91%, so it is right." | Check every value against something else. | A ten-times-too-large refund goes through. |
| Using a specialist for judgment | "A detector found the box, so it is fine." | Ask question 1 first: does it need meaning? | Damaged parcels are marked as fine. |
| Sending full-size images by default | "More detail is always better." | Crop or shrink to what the job needs. | You pay for pixels that do not help. |
| Thinking the model is the product | "A better model will fix it." | Build the pipeline around it. | The same mistakes come back with the new model. |
Check Your Understanding
Exercises
These exercises use new jobs, not the ten from the lab.
What to Remember
- A model that can see is not a model that is right.
- Images, pages and audio become tokens, and every token costs money and time.
- The multimodal LLM is for meaning, judgment and handwriting.
- Specialists are for exact text, plain speech, and finding or counting things.
- Silent errors (wrong and confident) are the most dangerous.
- Ask four questions in order: meaning? speech? things? text?
- The model is one stage of six. You write Ingest, Structure, Verify and Route.
Further Reading
- Transformers.js documentation, by Hugging Face: run vision, speech and text models in the browser.
- How to build a real-time voice AI agent in the browser, by LogRocket: a speech pipeline that runs fully on the device.
Next: Images In: Resolution, Crops and Token Cost, where you find the cheapest image size that still reads every receipt total correctly.