APVR: Hour-Level Long Video Understanding with Adaptive Pivot Visual Information Retrieval

TL;DR

APVR is a training-free framework that hierarchically retrieves and retains important visual information from hour-long videos, enabling large language models to better understand long videos without heavy resource demands.

Contribution

The paper introduces APVR, a novel hierarchical retrieval method that overcomes memory and resource limitations for long video understanding in multimodal large language models.

Findings

Achieves up to 9.7% performance improvement on LongVideoBench.

Demonstrates state-of-the-art results for training-free and training-based methods.

Effectively processes hour-long videos with maintained semantic fidelity.

Abstract

Current multimodal large language models (MLLMs) struggle with hour-level video understanding, facing significant challenges not only in modeling the substantial information volume of long videos but also in overcoming the memory wall and resource constraints during both training and inference. Although recent training-free approaches have alleviated resource demands by compressing visual features, their reliance on incomplete visual information limits the performance potential. To address these limitations, we propose Adaptive Pivot Visual information Retrieval (APVR), a training-free framework that hierarchically retrieves and retains sufficient and important visual information. It breakthroughs the memory wall limitation via two complementary components: Pivot Frame Retrieval employs query expansion and iterative spatio-semantic confidence scoring to identify relevant video frames, and Pivot Token Retrieval performs query-aware attention-driven token selection within up to 1024 pivot frames. This dual granularity approach enables the processing of hour-long videos while maintaining semantic fidelity. Experimental validations on three different baseline MLLMs demonstrate significant performance improvements up to 9.5\%, 4.6\% and 9.7\% on LongVideoBench, VideoMME and MLVU, respectively. APVR achieves state-of-the-art results for both training-free and training-based approaches.