# Video Understanding

Course: [Multimodal AI](https://www.devtutorials.ai/learn/multimodal-ai) · Section: Listening, Speaking and Generating
Source: https://www.devtutorials.ai/learn/multimodal-ai/listening-and-generating/video-understanding

> Find short moments in long clips without sending every frame: frame budgets, two passes around motion and sound, key frames for long video, and video-native models.

## What Video Understanding Is

**Video understanding** = getting a model to answer questions about a video: what happened, and when.

In this lesson, the claims app reads delivery-camera clips. A customer says their parcel was damaged, and the app looks for the moment it happened.

Here is the catch: most multimodal models do not watch video. They look at a few still pictures, called **frames**, taken from it. If the important moment falls between two frames, the model never sees it.

Press **Play**, or use **Next** to go one step at a time.

- **Frame 1:** a 90-second clip. A box falls off a trolley at 31.2 s, for 1.2 seconds.
- **Frame 2:** the app sends one frame every 5 seconds. None of them shows the fall. The model says nothing happened.

**Simple lesson:** a model only knows the frames you send it.

### A Real-World Example

Think about a security guard checking a night of camera footage.

- They cannot watch eight hours at normal speed.
- They fast-forward, and slow down when something moves.
- They watch the moment closely, then note the time.

A video pipeline does the same:

- **Fast-forward** = a few coarse frames for the whole clip.
- **Slow down when something moves** = a cheap motion detector finds where to look.
- **Watch closely** = many frames just around that moment.
- **Turn up the sound** = a loudness detector, for things you hear but cannot see.
- **Note the time** = return the timestamp with the answer.

**Simple lesson:** look quickly everywhere, then closely where it matters.

## What You Will Learn

By the end of this lesson, you will be able to:

- explain why models read frames, not video
- count how many frames a sampling rate sends
- compare uniform, scene-change and two-pass sampling
- use the sound track as a second signal
- handle long clips with key frames
- compare your own sampling with a video-native model
- stay within a frame budget, and return a timestamp

## Frames Cost Money

Each frame is an image, and images cost tokens (Lesson 2). **Frames per second (fps)** sets how many frames you send.

frames = seconds × fps, rounded up.

Step through all four frames.

- **The bar:** frames sent for the longest clip (150 s).
- **The score:** how many of the 4 moments are seen, and whether the clip is within budget.

| Sampling | Frames for 150 s | Sees a 1 s event? |
|---|---|---|
| 0.2 fps (every 5 s) | 30 | Often not |
| 0.5 fps (every 2 s) | 75 | Sometimes |
| 1 fps | 150 | Usually |
| 2 fps | 300 | Yes, but far over budget |

The lab's **frame budget** is 40 frames per clip. The lab has six clips, from 60 seconds to 15 minutes.

**Scene changes are not events.** Sampling at scene changes sends a frame when the picture changes a lot, like a camera switch. A box falling in the same scene often does not count as a scene change, so it is missed.

**Simple lesson:** uniform sampling trades cost against misses. There is no good rate for every clip.

## Two Passes: Find, Then Look Closely

Step through both frames.

- **Pass 1:** one frame every 10 seconds, plus a motion detector. It compares neighbouring frames with simple maths; no model is needed.
- **Pass 2:** 4 frames per second for 1 second either side of each motion spike. The model finds the fall at the right time, with about 30 frames in total.

## Listen Too

Video has a sound track. Some events are heard, not seen.

Step through both frames.

- **Frame 1:** a parcel is thrown over a garden gate and lands out of view. Motion finds nothing.
- **Frame 2:** a loudness detector marks the thud. Dense frames there find it.

**Some events have neither motion nor sound.** A parcel left in the rain for an hour does not move or make a noise. For those, sample regularly, or ask a person to check.

## Long Clips

Cost grows with the length of the clip, not with how much happens.

Step through both frames.

- **Frame 1:** a 15-minute warehouse clip. Even one frame every 10 s is over budget.
- **Frame 2:** one **key frame** per minute, plus the free detectors. 39 frames, and the moment is found.

Keep a one-line text summary for each minute's key frame. Search (Lesson 12) can then find moments across hours of video.

## Video-Native Models

Some models accept a video file directly, for example Gemini. They still sample frames inside, often about 1 per second, and bill you for every one. That is simple, and good for short clips. For long clips, choosing the frames yourself is far cheaper.

| Approach | Good for | Watch out for |
|---|---|---|
| Send the whole video | Short clips, quick prototypes | About 1 frame per second, billed |
| Two passes: detectors, then dense frames | Long clips, rare events | Events with no motion or sound |
| Key frames per minute | Hours of footage, search | Short events between key frames |

**Easy way to remember:** cropping in Lesson 2 zoomed in on space. Two-pass sampling zooms in on time.

Click the button and look at the **Pipeline** tab.

- **The player:** press **Play the moment** to watch it, and hear the sound (made in your browser).
- **The lanes:** frames sent, motion spikes and sound spikes, along the clip.
- **Frames near the moment:** the actual frames the model would see. A highlighted frame shows the event.

### Lab Exercise: See Every Moment Within Budget

Your task: change the **Ingest** settings (which frames go in), then press **Check**.

Click the button and open the **Pipeline** tab.

- **Which frames go in:** every N seconds, scene changes, two passes, or the whole video to a video model.
- **Listen too, and key frames:** for the two-pass strategy.
- **To pass:** all 6 moments found within 2 seconds, and no clip over 40 frames.

## Build It in TypeScript

```ts
export function sampleFrames({ seconds, motionAt, soundAt }: ClipInfo): number[] {
  const frames = new Set<number>();
  for (let t = 0; t < seconds; t += 60) frames.add(t);                   // one key frame per minute
  for (const spike of [...motionAt, ...soundAt])                          // look closely where something moved or made a noise
    for (let t = spike - 1; t < spike + 1; t += 0.25) if (t >= 0 && t < seconds) frames.add(Math.round(t * 100) / 100);
  return [...frames].sort((a, b) => a - b);
}
```

Click the button and open the **Code** tab.

- **The starter:** one frame every 5 seconds.
- **Run on the media bank:** your function runs on all 6 clips.

## TL;DR

1. Models read **frames**, not video.
2. **frames = seconds × fps.** Each frame costs tokens.
3. **Uniform** sampling misses short events or goes over budget.
4. **Scene changes** are not events.
5. **Two passes:** free detectors (motion and **sound**) find where to look; dense frames there.
6. **Long clips:** one key frame per minute, plus the detectors.
7. Return a **timestamp** with every answer.

## Common mistakes

| Mistake | Wrong idea | Correct idea | What goes wrong |
|---|---|---|---|
| Too few frames | "Every 5 s is plenty." | Look closely where something moves. | The fall happens between frames. |
| Too many frames | "Send every second." | Stay within a frame budget. | Each clip costs five times as much. |
| Trusting "nothing happened" | "The model watched it." | The model saw a few frames. | A real claim is refused. |
| Scene changes only | "Changes mean events." | Use motion, not scene cuts. | Events in one scene are missed. |
| Ignoring the sound | "Video is pictures." | Use loudness as a second signal. | A parcel heard but not seen is missed. |
| Sending long video whole | "The model handles it." | Key frames plus detectors. | 900 frames billed for one event. |
| No timestamp | "Yes or no is enough." | Return when it happened. | A reviewer must watch the whole clip. |

## Check Your Understanding

## Exercises

## What to Remember

- A model knows only the frames you send.
- More frames cost more.
- Find where to look cheaply (motion and sound), then look closely.
- Return a timestamp.
- "Nothing happened" may mean "nothing was seen".

## Further Reading

- [Gemini API: video understanding, by Google](https://ai.google.dev/gemini-api/docs/video-understanding): how a hosted model samples frames from video.
- [OpenCV: background subtraction](https://docs.opencv.org/4.x/d1/dc5/tutorial_background_subtraction.html): a simple way to detect motion.

Next: **Video Generation**, where the app makes short product videos, and checks every frame before they are used.