Listening, Speaking and Generating in Multimodal AI

Lessons in this section

1. Speech to Text

   Transcribe voice notes the app can trust: vocabulary hints for product names, number normalisation, word error rate, and timestamps.

2. Voice Agents and Turn-Taking

   Build a voice agent that answers fast without talking over people: streaming, endpointing, barge-in, noise handling, holding lines during tool calls, and hand-over to a person.

3. Image Generation with Checks

   Run an image studio: write the prompt, choose a model, generate candidates, check them, repair small flaws with edits, get a person's approval and attach a C2PA label.

4. Video Understanding

   Find short moments in long clips without sending every frame: frame budgets, two passes around motion and sound, key frames for long video, and video-native models.

5. Video Generation

   Generate short product videos that stay on-model: image-to-video from the product photo, several takes, frame checks for identity, words and motion, cost per second, and a visible AI label.

6. Multimodal Search

   Search a document archive by words and by looks: OCR text and page-image embeddings, routing, re-ranking, and answers that cite their pages.

7. Cost, Privacy and On-Device

   Keep personal data on the device: run OCR and speech locally, send text not photos, redact text, blur faces and plates, keep data only as long as needed, and delete it on request.

8. Capstone: Ship the Claims Pipeline

   Run every lesson's settings on a held-out set, pass five axes (accuracy, verification, cost, latency and privacy), and export the full pipeline.