Multimodal AI tutorial

Build features that see, read, listen and speak: receipts, invoices, screenshots, charts, voice and video. Pick the right model, check what it extracts, and know what each choice costs.

Start the course

Course contents

Seeing and Reading

  1. What Multimodal Models Can and Can't Do — A model that can see is not a model that is right. Run ten claim items through a multimodal LLM and through specialist models, compare accuracy, cost and time, and learn four questions that pick the right tool.
  2. Images In: Resolution, Crops and Token Cost — Every pixel costs tokens, but shrinking a photo can erase the one number you need. Find the cheapest way to send twelve receipts that still reads every total, with resizing and crops.
  3. Documents: OCR, Layout, Tables and Forms — Invoices break extraction in four ways: merged cells shift columns, long tables get cut short, dates are read the wrong way round, and money stays as text. Fix each one, and extract ten invoices to a schema.
  4. Screenshots, UI and Charts — Turn error screenshots into complete bug reports and read charts honestly. Small error codes need OCR, summaries need a schema, and values read from bar heights must be flagged as estimates.
  5. Verification and Grounding — A model that sounds sure can still be wrong. Add cross-checks, format and range checks, grounding to a box on the page, and a confidence threshold, then route doubtful fields to a person.
  6. Human Review and Your v1 Pipeline — Build the review screen: each doubtful field beside its source region, one click to accept or correct, and every correction saved as an Evals row. Then export your v1 document pipeline.

Listening, Speaking and Generating

  1. Speech to Text — Transcribe voice notes the app can trust: vocabulary hints for product names, number normalisation, word error rate, and timestamps.
  2. Voice Agents and Turn-Taking — Build a voice agent that answers fast without talking over people: streaming, endpointing, barge-in, noise handling, holding lines during tool calls, and hand-over to a person.
  3. Image Generation with Checks — Run an image studio: write the prompt, choose a model, generate candidates, check them, repair small flaws with edits, get a person's approval and attach a C2PA label.
  4. Video Understanding — Find short moments in long clips without sending every frame: frame budgets, two passes around motion and sound, key frames for long video, and video-native models.
  5. Video Generation — Generate short product videos that stay on-model: image-to-video from the product photo, several takes, frame checks for identity, words and motion, cost per second, and a visible AI label.
  6. Multimodal Search — Search a document archive by words and by looks: OCR text and page-image embeddings, routing, re-ranking, and answers that cite their pages.
  7. Cost, Privacy and On-Device — Keep personal data on the device: run OCR and speech locally, send text not photos, redact text, blur faces and plates, keep data only as long as needed, and delete it on request.
  8. Capstone: Ship the Claims Pipeline — Run every lesson's settings on a held-out set, pass five axes (accuracy, verification, cost, latency and privacy), and export the full pipeline.

Reference

  1. Multimodal AI Cheat Sheet — Every rule from the course on one page: tool choice, image sizing, documents, verification, review, speech, voice agents, the image studio, video understanding and generation, search, privacy and every lab target.