StreamKV: Streaming Video Question-Answering with Segment-based KV Cache Retrieval and Compression

TL;DR

StreamKV introduces a dynamic, segment-based KV cache retrieval and compression framework for Video-LLMs, significantly enhancing accuracy, memory efficiency, and latency in streaming video question answering.

Contribution

It proposes a training-free, unified approach that dynamically partitions videos, summarizes segments, and compresses KV caches to improve streaming Video-LLMs performance.

Findings

Outperforms existing Online Video-LLMs on StreamingVQA benchmarks.

Achieves higher accuracy with lower memory and computational costs.

Effectively preserves semantic information through segment-based processing.

Abstract

Video Large Language Models (Video-LLMs) have demonstrated significant potential in the areas of video captioning, search, and summarization. However, current Video-LLMs still face challenges with long real-world videos. Recent methods have introduced a retrieval mechanism that retrieves query-relevant KV caches for question answering, enhancing the efficiency and accuracy of long real-world videos. However, the compression and retrieval of KV caches are still not fully explored. In this paper, we propose \textbf{StreamKV}, a training-free framework that seamlessly equips Video-LLMs with advanced KV cache retrieval and compression. Compared to previous methods that used uniform partitioning, StreamKV dynamically partitions video streams into semantic segments, which better preserves semantic information. For KV cache retrieval, StreamKV calculates a summary vector for each segment to retain segment-level information essential for retrieval. For KV cache compression, StreamKV introduces a guidance prompt designed to capture the key semantic elements within each segment, ensuring only the most informative KV caches are retained for answering questions. Moreover, StreamKV unifies KV cache retrieval and compression within a single module, performing both in a layer-adaptive manner, thereby further improving the effectiveness of streaming video question answering. Extensive experiments on public StreamingVQA benchmarks demonstrate that StreamKV significantly outperforms existing Online Video-LLMs, achieving superior accuracy while substantially improving both memory efficiency and computational latency. The code has been released at https://github.com/sou1p0wer/StreamKV.