Shot-Aware Frame Sampling for Video Understanding

TL;DR

This paper introduces InfoShot, a shot-aware frame sampling method that enhances long-video understanding by selecting keyframes to preserve both overall context and critical short events, improving performance without retraining.

Contribution

The paper proposes a novel, task-agnostic shot-aware sampling technique that partitions videos into shots and selects representative frames based on information theory, improving the capture of important short events.

Findings

Improves anomaly detection hit rate and Video-QA accuracy under frame constraints.

Outperforms strong baselines on standard video understanding benchmarks.

Introduces SynFlash, a synthetic benchmark for short-lived event detection.

Abstract

Video frame sampling is essential for efficient long-video understanding with Vision-Language Models (VLMs), since dense inputs are costly and often exceed context limits. Yet when only a small number of frames can be retained, existing samplers often fail to balance broad video coverage with brief but critical events, which can lead to unreliable downstream predictions. To address this issue, we present InfoShot, a task-agnostic, shot-aware frame sampler for long-video understanding. InfoShot first partitions a video into semantically consistent shots, and then selects two complementary keyframes from each shot: one to represent the main content and one to capture unusual within-shot changes. This design is guided by an information-theoretic objective that encourages the sampled set to retain high information about both shot structure and sparse within-shot deviations. In this way, it improves the chance of preserving both overall video context and short decision-critical moments without requiring any retraining. To better evaluate such short-lived events, we further introduce SynFlash, a synthetic benchmark with controllable sub-second anomaly patterns and frame-level ground truth, and we also evaluate InfoShot on existing anomaly datasets and general video understanding tasks. Experiments show that InfoShot improves anomaly hit rate and downstream Video-QA accuracy under frame number constraints, while matching or outperforming strong baselines on standard video understanding benchmarks.