Attend Before Attention: Efficient and Scalable Video Understanding via Autoregressive Gazing

TL;DR

AutoGaze is a novel module that efficiently reduces redundant visual information in long videos, enabling scalable high-resolution video understanding with significant speedups and improved benchmark performance.

Contribution

We propose AutoGaze, a lightweight autoregressive patch selection method that drastically reduces computational load while maintaining video information for large-scale video understanding.

Findings

AutoGaze reduces visual tokens by 4x-100x.

AutoGaze accelerates ViTs and MLLMs by up to 19x.

AutoGaze enables scaling to 1K-frame 4K videos and improves benchmark results.

Abstract

Multi-modal large language models (MLLMs) have advanced general-purpose video understanding but struggle with long, high-resolution videos -- they process every pixel equally in their vision transformers (ViTs) or LLMs despite significant spatiotemporal redundancy. We introduce AutoGaze, a lightweight module that removes redundant patches before processed by a ViT or an MLLM. Trained with next-token prediction and reinforcement learning, AutoGaze autoregressively selects a minimal set of multi-scale patches that can reconstruct the video within a user-specified error threshold, eliminating redundancy while preserving information. Empirically, AutoGaze reduces visual tokens by 4x-100x and accelerates ViTs and MLLMs by up to 19x, enabling scaling MLLMs to 1K-frame 4K-resolution videos and achieving superior results on video benchmarks (e.g., 67.0% on VideoMME). Furthermore, we introduce HLVid: the first high-resolution, long-form video QA benchmark with 5-minute 4K-resolution videos, where an MLLM scaled with AutoGaze improves over the baseline by 10.1% and outperforms the previous best MLLM by 4.5%. Project page: https://autogaze.github.io/.