Video Understanding

By the Syncfusion DevTutorials team · Last updated October 27, 2026

Find short moments in long clips without sending every frame: frame budgets, two passes around motion and sound, key frames for long video, and video-native models.

What Video Understanding Is

Video understanding = getting a model to answer questions about a video: what happened, and when.

In this lesson, the claims app reads delivery-camera clips. A customer says their parcel was damaged, and the app looks for the moment it happened.

Here is the catch: most multimodal models do not watch video. They look at a few still pictures, called frames, taken from it. If the important moment falls between two frames, the model never sees it.

Press Play, or use Next to go one step at a time.

• Frame 1: a 90-second clip. A box falls off a trolley at 31.2 s, for 1.2 seconds.
• Frame 2: the app sends one frame every 5 seconds. None of them shows the fall. The model says nothing happened.

Simple lesson: a model only knows the frames you send it.

A Real-World Example

Think about a security guard checking a night of camera footage.

• They cannot watch eight hours at normal speed.
• They fast-forward, and slow down when something moves.
• They watch the moment closely, then note the time.

A video pipeline does the same:

• Fast-forward = a few coarse frames for the whole clip.
• Slow down when something moves = a cheap motion detector finds where to look.
• Watch closely = many frames just around that moment.
• Turn up the sound = a loudness detector, for things you hear but cannot see.
• Note the time = return the timestamp with the answer.

Simple lesson: look quickly everywhere, then closely where it matters.

What You Will Learn

By the end of this lesson, you will be able to:

• explain why models read frames, not video
• count how many frames a sampling rate sends
• compare uniform, scene-change and two-pass sampling
• use the sound track as a second signal
• handle long clips with key frames
• compare your own sampling with a video-native model
• stay within a frame budget, and return a timestamp

Frames Cost Money

Each frame is an image, and images cost tokens (Lesson 2). Frames per second (fps) sets how many frames you send.

frames = seconds × fps, rounded up.

Step through all four frames.

• The bar: frames sent for the longest clip (150 s).
• The score: how many of the 4 moments are seen, and whether the clip is within budget.

The lab's frame budget is 40 frames per clip. The lab has six clips, from 60 seconds to 15 minutes.

Simple lesson: uniform sampling trades cost against misses. There is no good rate for every clip.

Two Passes: Find, Then Look Closely

Step through both frames.

• Pass 1: one frame every 10 seconds, plus a motion detector. It compares neighbouring frames with simple maths; no model is needed.
• Pass 2: 4 frames per second for 1 second either side of each motion spike. The model finds the fall at the right time, with about 30 frames in total.

Listen Too

Video has a sound track. Some events are heard, not seen.

Step through both frames.

• Frame 1: a parcel is thrown over a garden gate and lands out of view. Motion finds nothing.
• Frame 2: a loudness detector marks the thud. Dense frames there find it.

Long Clips

Cost grows with the length of the clip, not with how much happens.

Step through both frames.

• Frame 1: a 15-minute warehouse clip. Even one frame every 10 s is over budget.
• Frame 2: one key frame per minute, plus the free detectors. 39 frames, and the moment is found.

Keep a one-line text summary for each minute's key frame. Search (Lesson 12) can then find moments across hours of video.

Video-Native Models

Some models accept a video file directly, for example Gemini. They still sample frames inside, often about 1 per second, and bill you for every one. That is simple, and good for short clips. For long clips, choosing the frames yourself is far cheaper.

Click the button and look at the Pipeline tab.

• The player: press Play the moment to watch it, and hear the sound (made in your browser).
• The lanes: frames sent, motion spikes and sound spikes, along the clip.
• Frames near the moment: the actual frames the model would see. A highlighted frame shows the event.

Lab Exercise: See Every Moment Within Budget

Your task: change the Ingest settings (which frames go in), then press Check.

Click the button and open the Pipeline tab.

• Which frames go in: every N seconds, scene changes, two passes, or the whole video to a video model.
• Listen too, and key frames: for the two-pass strategy.
• To pass: all 6 moments found within 2 seconds, and no clip over 40 frames.

Build It in TypeScript

export function sampleFrames({ seconds, motionAt, soundAt }: ClipInfo): number[] {
  const frames = new Set<number>();
  for (let t = 0; t < seconds; t += 60) frames.add(t);                   // one key frame per minute
  for (const spike of [...motionAt, ...soundAt])                          // look closely where something moved or made a noise
    for (let t = spike - 1; t < spike + 1; t += 0.25) if (t >= 0 && t < seconds) frames.add(Math.round(t * 100) / 100);
  return [...frames].sort((a, b) => a - b);
}

Click the button and open the Code tab.

• The starter: one frame every 5 seconds.
• Run on the media bank: your function runs on all 6 clips.

TL;DR

1. Models read frames, not video.
2. frames = seconds × fps. Each frame costs tokens.
3. Uniform sampling misses short events or goes over budget.
4. Scene changes are not events.
5. Two passes: free detectors (motion and sound) find where to look; dense frames there.
6. Long clips: one key frame per minute, plus the detectors.
7. Return a timestamp with every answer.

Common mistakes

Check Your Understanding

Exercises

What to Remember

• A model knows only the frames you send.
• More frames cost more.
• Find where to look cheaply (motion and sound), then look closely.
• Return a timestamp.
• "Nothing happened" may mean "nothing was seen".

Further Reading

• Gemini API: video understanding, by Google: how a hosted model samples frames from video.
• OpenCV: background subtraction: a simple way to detect motion.

Next: Video Generation, where the app makes short product videos, and checks every frame before they are used.