Video-XL-2: Towards Very Long-Video Understanding Through Task-Aware KV Sparsification

TL;DR

Video-XL-2 introduces a task-aware key-value sparsification method that significantly improves the efficiency and performance of long-video understanding in multi-modal large language models, enabling processing of thousands of frames with reduced computational costs.

Contribution

The paper presents a novel framework combining chunk-based pre-filling and bi-level key-value decoding to enhance long-video understanding efficiency and accuracy.

Findings

Achieves state-of-the-art results on long video benchmarks.

Processes over 10,000 frames on a single GPU.

Outperforms existing lightweight models in efficiency and accuracy.

Abstract

Multi-modal large language models (MLLMs) models have made significant progress in video understanding over the past few years. However, processing long video inputs remains a major challenge due to high memory and computational costs. This makes it difficult for current models to achieve both strong performance and high efficiency in long video understanding. To address this challenge, we propose Video-XL-2, a novel MLLM that delivers superior cost-effectiveness for long-video understanding based on task-aware KV sparsification. The proposed framework operates with two key steps: chunk-based pre-filling and bi-level key-value decoding. Chunk-based pre-filling divides the visual token sequence into chunks, applying full attention within each chunk and sparse attention across chunks. This significantly reduces computational and memory overhead. During decoding, bi-level key-value decoding selectively reloads either dense or sparse key-values for each chunk based on its relevance to the task. This approach further improves memory efficiency and enhances the model's ability to capture fine-grained information. Video-XL-2 achieves state-of-the-art performance on various long video understanding benchmarks, outperforming existing open-source lightweight models. It also demonstrates exceptional efficiency, capable of processing over 10,000 frames on a single NVIDIA A100 (80GB) GPU and thousands of frames in just a few seconds.