Venus: An Efficient Edge Memory-and-Retrieval System for VLM-based Online Video Understanding

TL;DR

Venus is an edge-cloud system that enables efficient, real-time video understanding by hierarchical memory management and adaptive keyframe retrieval, significantly reducing latency while maintaining high reasoning accuracy.

Contribution

Venus introduces a novel edge-cloud architecture with hierarchical memory and adaptive retrieval for efficient online video understanding on resource-constrained devices.

Findings

Achieves 15x-131x speedup in response latency

Maintains comparable or superior reasoning accuracy

Enables real-time video understanding on edge devices

Abstract

Vision-language models (VLMs) have demonstrated impressive multimodal comprehension capabilities and are being deployed in an increasing number of online video understanding applications. While recent efforts extensively explore advancing VLMs' reasoning power in these cases, deployment constraints are overlooked, leading to overwhelming system overhead in real-world deployments. To address that, we propose Venus, an on-device memory-and-retrieval system for efficient online video understanding. Venus proposes an edge-cloud disaggregated architecture that sinks memory construction and keyframe retrieval from cloud to edge, operating in two stages. In the ingestion stage, Venus continuously processes streaming edge videos via scene segmentation and clustering, where the selected keyframes are embedded with a multimodal embedding model to build a hierarchical memory for efficient storage and retrieval. In the querying stage, Venus indexes incoming queries from memory, and employs a threshold-based progressive sampling algorithm for keyframe selection that enhances diversity and adaptively balances system cost and reasoning accuracy. Our extensive evaluation shows that Venus achieves a 15x-131x speedup in total response latency compared to state-of-the-art methods, enabling real-time responses within seconds while maintaining comparable or even superior reasoning accuracy.