IQViC: In-context, Question Adaptive Vision Compressor for Long-term Video Understanding LMMs

TL;DR

IQViC introduces a transformer-based visual compressor that enhances long-term video understanding by selectively compressing relevant information conditioned on questions, improving accuracy and memory efficiency.

Contribution

The paper proposes IQViC, a novel question-adaptive visual compressor that enables efficient long-term video understanding by reducing memory requirements and focusing on relevant content.

Findings

Outperforms existing methods in accuracy on long-term video QA tasks.

Reduces memory token usage significantly compared to full feature methods.

Effective on a new InfiniBench-based dataset and standard benchmarks.

Abstract

With the increasing complexity of video data and the need for more efficient long-term temporal understanding, existing long-term video understanding methods often fail to accurately capture and analyze extended video sequences. These methods typically struggle to maintain performance over longer durations and to handle the intricate dependencies within the video content. To address these limitations, we propose a simple yet effective large multi-modal model framework for long-term video understanding that incorporates a novel visual compressor, the In-context, Question Adaptive Visual Compressor (IQViC). The key idea, inspired by humans' selective attention and in-context memory mechanisms, is to introduce a novel visual compressor and incorporate efficient memory management techniques to enhance long-term video question answering. Our framework utilizes IQViC, a transformer-based visual compressor, enabling question-conditioned in-context compression, unlike existing methods that rely on full video visual features. This selectively extracts relevant information, significantly reducing memory token requirements. Through extensive experiments on a new dataset based on InfiniBench for long-term video understanding, and standard benchmarks used for existing methods' evaluation, we demonstrate the effectiveness of our proposed IQViC framework and its superiority over state-of-the-art methods in terms of video understanding accuracy and memory efficiency.