Vista: Scene-Aware Optimization for Streaming Video Question Answering under Post-Hoc Queries

TL;DR

Vista introduces a scene-aware streaming video QA framework that dynamically segments, compresses, and recalls video scenes for efficient, scalable, and accurate reasoning over continuous video streams in real-time scenarios.

Contribution

The paper proposes a novel scene-aware framework for streaming video QA that improves efficiency and scalability by dynamic segmentation, compression, and selective recall of video scenes.

Findings

Achieves state-of-the-art performance on StreamingBench.

Enables long-context reasoning without latency or memory issues.

Demonstrates seamless integration with various vision-language models.

Abstract

Streaming video question answering (Streaming Video QA) poses distinct challenges for multimodal large language models (MLLMs), as video frames arrive sequentially and user queries can be issued at arbitrary time points. Existing solutions relying on fixed-size memory or naive compression often suffer from context loss or memory overflow, limiting their effectiveness in long-form, real-time scenarios. We present Vista, a novel framework for scene-aware streaming video QA that enables efficient and scalable reasoning over continuous video streams. The innovation of Vista can be summarized in three aspects: (1) scene-aware segmentation, where Vista dynamically clusters incoming frames into temporally and visually coherent scene units; (2) scene-aware compression, where each scene is compressed into a compact token representation and stored in GPU memory for efficient index-based retrieval, while full-resolution frames are offloaded to CPU memory; and (3) scene-aware recall, where relevant scenes are selectively recalled and reintegrated into the model input upon receiving a query, enabling both efficiency and completeness. Vista is model-agnostic and integrates seamlessly with a variety of vision-language backbones, enabling long-context reasoning without compromising latency or memory efficiency. Extensive experiments on StreamingBench demonstrate that Vista achieves state-of-the-art performance, establishing a strong baseline for real-world streaming video understanding.