LiveVLM: Efficient Online Video Understanding via Streaming-Oriented KV Cache and Retrieval

TL;DR

LiveVLM introduces a real-time, training-free framework for online video understanding that reduces memory and delay issues in Video LLMs through innovative KV cache management and retrieval mechanisms.

Contribution

It proposes LiveVLM, a novel framework combining Vision Sink Bucketing and Position-agnostic KV Retrieval for efficient, online, query-agnostic video processing without additional training.

Findings

Achieves state-of-the-art accuracy among training-free query-agnostic methods.

Reduces memory overhead and response delay in online video understanding.

Supports real-time interaction in applications like autonomous driving and robotics.

Abstract

Recent developments in Video Large Language Models (Video LLMs) have enabled models to process hour-long videos and exhibit exceptional performance. Nonetheless, the Key-Value (KV) cache expands linearly over time, leading to substantial memory overhead and response delay--critical challenges in various real-world online applications, such as Deepseek services, autonomous driving and robotics. To mitigate these issues, we propose $\textbf{LiveVLM}$, a training-free and query-agnostic framework specifically designed for online video understanding and real-time interaction. LiveVLM employs a Vision Sink Bucketing (VSB) mechanism to process video streams in real time, retain long-term video details and eliminate redundant KVs. This mechanism utilizes vision-to-vision attention scores as the metric and seeks to maximize the coverage of contextual information during compression. Noting that KV cache compressed in a query-agnostic manner inevitably retains irrelevant information for specific queries, LiveVLM incorporates a Position-agnostic KV Retrieval (PaR) mechanism to reduce interference from redundant context. The keypoint of PaR lies in decoupling positional embeddings to enhance the similarity between key tensors, thereby supporting efficient retrieval at the granularity of pages. Extensive experiments demonstrate that LiveVLM enables the foundation LLaVA-OneVision model to achieve state-of-the-art accuracy among both training-free query-agnostic methods and training-based online models.