HFS: Holistic Query-Aware Frame Selection for Efficient Video Reasoning

TL;DR

This paper introduces an end-to-end trainable, task-adaptive frame selection framework for video reasoning that dynamically optimizes frame relevance, coverage, and redundancy, outperforming existing methods across multiple benchmarks.

Contribution

It proposes a novel holistic, query-aware frame selection method using a Chain-of-Thought guided language model and differentiable set-level optimization, eliminating reliance on static pseudo labels.

Findings

Significantly outperforms existing methods on multiple benchmarks.

Effectively balances relevance, coverage, and redundancy in frame selection.

Enables dynamic, task-specific frame selection through end-to-end training.

Abstract

Key frame selection in video understanding presents significant challenges. Traditional top-K selection methods, which score frames independently, often fail to optimize the selection as a whole. This independent scoring frequently results in selecting frames that are temporally clustered and visually redundant. Additionally, training lightweight selectors using pseudo labels generated offline by Multimodal Large Language Models (MLLMs) prevents the supervisory signal from dynamically adapting to task objectives. To address these limitations, we propose an end-to-end trainable, task-adaptive framework for frame selection. A Chain-of-Thought approach guides a Small Language Model (SLM) to generate task-specific implicit query vectors, which are combined with multimodal features to enable dynamic frame scoring. We further define a continuous set-level objective function that incorporates relevance, coverage, and redundancy, enabling differentiable optimization via Gumbel-Softmax to select optimal frame combinations at the set level. Finally, student-teacher mutual learning is employed, where the student selector (SLM) and teacher reasoner (MLLM) are trained to align their frame importance distributions via KL divergence. Combined with cross-entropy loss, this enables end-to-end optimization, eliminating reliance on static pseudo labels. Experiments across various benchmarks, including Video-MME, LongVideoBench, MLVU, and NExT-QA, demonstrate that our method significantly outperforms existing approaches.