VideoBrain: Learning Adaptive Frame Sampling for Long Video Understanding

TL;DR

VideoBrain introduces an adaptive frame sampling framework for long video understanding, enabling vision-language models to efficiently select informative frames through learned policies, improving accuracy while reducing computational costs.

Contribution

It presents a novel end-to-end adaptive sampling method with dual agents and a behavior-aware reward, enhancing long video comprehension over prior uniform or single-pass sampling approaches.

Findings

Achieves 3.5% to 9.0% accuracy improvements on benchmarks.

Uses 30-40% fewer frames than baseline methods.

Demonstrates strong generalization across datasets.

Abstract

Long-form video understanding remains challenging for Vision-Language Models (VLMs) due to the inherent tension between computational constraints and the need to capture information distributed across thousands of frames. Existing approaches either sample frames uniformly (risking information loss) or select keyframes in a single pass (with no recovery from poor choices). We propose VideoBrain, an end-to-end framework that enables VLMs to adaptively acquire visual information through learned sampling policies. Our approach features dual complementary agents: a CLIP-based agent for semantic retrieval across the video and a Uniform agent for dense temporal sampling within intervals. Unlike prior agent-based methods that rely on text-only LLMs orchestrating visual tools, our VLM directly perceives frames and reasons about information sufficiency. To prevent models from invoking agents indiscriminately to maximize rewards, we introduce a behavior-aware reward function coupled with a data classification pipeline that teaches the model when agent invocation is genuinely beneficial. Experiments on four long video benchmarks demonstrate that VideoBrain achieves +3.5% to +9.0% improvement over the baseline while using 30-40% fewer frames, with strong cross-dataset generalization to short video benchmarks.