Video Detective: Seek Critical Clues Recurrently to Answer Question from Long Videos

TL;DR

VideoDetective introduces a question-aware memory mechanism that iteratively compresses and aggregates critical clues from long videos, enabling efficient long video question-answering with limited context and reduced computational resources.

Contribution

The paper proposes a novel recurrent memory approach for long video question-answering, improving efficiency and accuracy by focusing on crucial information and introducing a new dataset for evaluation.

Findings

Enables processing of 100K tokens with only 32K context length

Achieves significant reduction in GPU memory usage (2 minutes, 37GB)

Outperforms existing methods on multiple long video benchmarks

Abstract

Long Video Question-Answering (LVQA) presents a significant challenge for Multi-modal Large Language Models (MLLMs) due to immense context and overloaded information, which could also lead to prohibitive memory consumption. While existing methods attempt to address these issues by reducing visual tokens or extending model's context length, they may miss useful information or take considerable computation. In fact, when answering given questions, only a small amount of crucial information is required. Therefore, we propose an efficient question-aware memory mechanism, enabling MLLMs to recurrently seek these critical clues. Our approach, named VideoDetective, simplifies this task by iteratively processing video sub-segments. For each sub-segment, a question-aware compression strategy is employed by introducing a few special memory tokens to achieve purposefully compression. This allows models to effectively seek critical clues while reducing visual tokens. Then, due to history context could have a significant impact, we recurrently aggregate and store these memory tokens to update history context, which would be reused for subsequent sub-segments. Furthermore, to more effectively measure model's long video understanding ability, we introduce GLVC (Grounding Long Video Clues), a long video question-answering dataset, which features grounding critical and concrete clues scattered throughout entire videos. Experimental results demonstrate our method enables MLLMs with limited context length of 32K to efficiently process 100K tokens (3600 frames, an hour-long video sampled at 1fps), requiring only 2 minutes and 37GB GPU memory usage. Evaluation results across multiple long video benchmarks illustrate our method can more effectively seek critical clues from massive information.