MovieChat+: Question-aware Sparse Memory for Long Video Question Answering

TL;DR

MovieChat+ introduces a novel memory mechanism based on the Atkinson-Shiffrin model, enabling zero-shot long video question answering without complex temporal modules, achieving state-of-the-art results on a new long video benchmark.

Contribution

The paper proposes MovieChat+, a memory-augmented approach that leverages pre-trained large language models for zero-shot long video understanding without additional trainable temporal modules.

Findings

Achieves state-of-the-art performance on long video understanding tasks.

Introduces the MovieChat-1K benchmark with extensive annotations.

Demonstrates effectiveness of memory mechanism in long video QA.

Abstract

Recently, integrating video foundation models and large language models to build a video understanding system can overcome the limitations of specific pre-defined vision tasks. Yet, existing methods either employ complex spatial-temporal modules or rely heavily on additional perception models to extract temporal features for video understanding, and they only perform well on short videos. For long videos, the computational complexity and memory costs associated with long-term temporal connections are significantly increased, posing additional challenges.Taking advantage of the Atkinson-Shiffrin memory model, with tokens in Transformers being employed as the carriers of memory in combination with our specially designed memory mechanism, we propose MovieChat to overcome these challenges. We lift pre-trained multi-modal large language models for understanding long videos without incorporating additional trainable temporal modules, employing a zero-shot approach. MovieChat achieves state-of-the-art performance in long video understanding, along with the released MovieChat-1K benchmark with 1K long video, 2K temporal grounding labels, and 14K manual annotations for validation of the effectiveness of our method. The code along with the dataset can be accessed via the following https://github.com/rese1f/MovieChat.