Flash-VStream: Efficient Real-Time Understanding for Long Video Streams

TL;DR

Flash-VStream is a novel real-time video understanding model designed for extremely long videos, utilizing a specialized memory system to reduce latency and improve efficiency while maintaining state-of-the-art performance.

Contribution

The paper introduces Flash-VStream, a new model with a dual-memory system that efficiently processes long videos in real time, addressing computational challenges of existing methods.

Findings

Achieves significant inference latency reduction.

Demonstrates state-of-the-art performance on multiple long video benchmarks.

Effectively handles extremely long videos with high efficiency.

Abstract

Benefiting from the advances in large language models and cross-modal alignment, existing multimodal large language models have achieved prominent performance in image and short video understanding. However, the understanding of long videos is still challenging, as their long-context nature results in significant computational and memory overhead. Most existing work treats long videos in the same way as short videos, which is inefficient for real-world applications and hard to generalize to even longer videos. To address these issues, we propose Flash-VStream, an efficient video language model capable of processing extremely long videos and responding to user queries in real time. Particularly, we design a Flash Memory module, containing a low-capacity context memory to aggregate long-context temporal information and model the distribution of information density, and a high-capacity augmentation memory to retrieve detailed spatial information based on this distribution. Compared to existing models, Flash-VStream achieves significant reductions in inference latency. Extensive experiments on long video benchmarks and comprehensive video benchmarks, i.e., EgoSchema, MLVU, LVBench, MVBench and Video-MME, demonstrate the state-of-the-art performance and outstanding efficiency of our method. Code is available at https://github.com/IVGSZ/Flash-VStream.