Towards Sparse Video Understanding and Reasoning

TL;DR

The paper introduces evise, a multi-round agent for video question answering that selectively samples informative frames, maintains a summary state, and supports reinforcement fine-tuning, leading to more efficient and accurate video reasoning.

Contribution

It proposes a novel sparse video reasoning method with a plug-and-play design and introduces EAGER, a new reward for reinforcement fine-tuning of vision-language models.

Findings

Improves accuracy on multiple VQA benchmarks.

Reduces frames, rounds, and prompt tokens needed.

Enables reinforcement fine-tuning with EAGER reward.

Abstract

We present \revise (\underline{Re}asoning with \underline{Vi}deo \underline{S}parsity), a multi-round agent for video question answering (VQA). Instead of uniformly sampling frames, \revise selects a small set of informative frames, maintains a summary-as-state across rounds, and stops early when confident. It supports proprietary vision-language models (VLMs) in a ``plug-and-play'' setting and enables reinforcement fine-tuning for open-source models. For fine-tuning, we introduce EAGER (Evidence-Adjusted Gain for Efficient Reasoning), an annotation-free reward with three terms: (1) Confidence gain: after new frames are added, we reward the increase in the log-odds gap between the correct option and the strongest alternative; (2) Summary sufficiency: at answer time we re-ask using only the last committed summary and reward success; (3) Correct-and-early stop: answering correctly within a small turn budget is rewarded. Across multiple VQA benchmarks, \revise improves accuracy while reducing frames, rounds, and prompt tokens, demonstrating practical sparse video reasoning.