Video Active Perception: Effective Inference-Time Long-Form Video Understanding with Vision-Language Models

TL;DR

This paper introduces Video Active Perception (VAP), a training-free method that improves long-form video question answering by selecting keyframes more effectively using active perception principles and a lightweight video generation model.

Contribution

VAP applies active perception theory to select keyframes in long-form videos, significantly enhancing efficiency and reasoning ability without additional training.

Findings

Achieves up to 5.6x frame efficiency over standard methods.

State-of-the-art zero-shot results on multiple long-form video QA datasets.

Effectively selects question-relevant keyframes, improving reasoning.

Abstract

Large vision-language models (VLMs) have advanced multimodal tasks such as video question answering (QA). However, VLMs face the challenge of selecting frames effectively and efficiently, as standard uniform sampling is expensive and performance may plateau. Inspired by active perception theory, which posits that models gain information by acquiring data that differs from their expectations, we introduce Video Active Perception (VAP), a training-free method to enhance long-form video QA using VLMs. Our approach treats keyframe selection as data acquisition in active perception and leverages a lightweight text-conditioned video generation model to represent prior world knowledge. Empirically, VAP achieves state-of-the-art zero-shot results on long-form or reasoning video QA datasets such as EgoSchema, NExT-QA, ActivityNet-QA, IntentQA, and CLEVRER, achieving an increase of up to 5.6 x frame efficiency by frames per question over standard GPT-4o, Gemini 1.5 Pro, and LLaVA-OV. Moreover, VAP shows stronger reasoning abilities than previous methods and effectively selects keyframes relevant to questions. These findings highlight the potential of leveraging active perception to improve the frame effectiveness and efficiency of long-form video QA.