Divide, then Ground: Adapting Frame Selection to Query Types for Long-Form Video Understanding

TL;DR

This paper introduces DIG, a query-type adaptive frame selection method for long-form video understanding with large multimodal models, improving efficiency and performance by tailoring strategies to global or localized queries.

Contribution

We propose a training-free, adaptive frame selection framework that distinguishes query types and optimizes frame sampling strategies accordingly, reducing computational costs.

Findings

DIG outperforms existing baselines across three benchmarks.

It maintains high performance even with 256 input frames.

Efficiently adapts to query types, improving long-form video understanding.

Abstract

The application of Large Multimodal Models (LMMs) to long-form video understanding is constrained by limited context lengths and the computationally prohibitive cost of processing dense video tokens. Consequently, recent research has focused on query-aware frame selection, methods that often incur significant computational overhead. This paper challenges the assumption that such complex search mechanisms are universally necessary. We first identify and validate a query typology distinguishing between global query and localized query. We demonstrate that while uniform sampling is both effective and efficient for global queries, localized queries indeed necessitate query-aware selection for optimal performance. Building on this insight, we propose DIG, a training-free frame selection framework that adapts its strategy based on the query type. Specifically,DIG employs efficient uniform sampling for global queries while activating a specialized pipeline to extract query-relevant frames for localized queries. Experiments on three long-form video understanding benchmarks demonstrate that DIG consistently outperforms existing baselines and robustly improves LMM performance, even when scaling the input frame count to 256.