SVFusion: A CPU-GPU Co-Processing Architecture for Large-Scale Real-Time Vector Search

TL;DR

SVFusion is a CPU-GPU co-processing framework that significantly improves real-time large-scale vector search performance by combining hierarchical indexing, workload-aware caching, and concurrency control, enabling faster and more efficient approximate nearest neighbor searches.

Contribution

The paper introduces SVFusion, a novel hybrid CPU-GPU architecture with hierarchical indexing and adaptive resource management for real-time vector search, addressing limitations of existing solutions.

Findings

20.9x higher throughput on average

1.3x to 50.7x lower latency

Maintains high recall under streaming workloads

Abstract

Approximate Nearest Neighbor Search (ANNS) underpins modern applications such as information retrieval and recommendation. With the rapid growth of vector data, efficient indexing for real-time vector search has become rudimentary. Existing CPU-based solutions support updates but suffer from low throughput, while GPU-accelerated systems deliver high performance but face challenges with dynamic updates and limited GPU memory, resulting in a critical performance gap for continuous, large-scale vector search requiring both accuracy and speed. In this paper, we present SVFusion, a GPU-CPU-disk collaborative framework for real-time vector search that bridges sophisticated GPU computation with online updates. SVFusion leverages a hierarchical vector index architecture that employs CPU-GPU co-processing, along with a workload-aware vector caching mechanism to maximize the efficiency of limited GPU memory. It further enhances performance through real-time coordination with CUDA multi-stream optimization and adaptive resource management, along with concurrency control that ensures data consistency under interleaved queries and updates. Empirical results demonstrate that SVFusion achieves significant improvements in query latency and throughput, exhibiting a 20.9x higher throughput on average and 1.3x to 50.7x lower latency compared to baseline methods, while maintaining high recall for large-scale datasets under various streaming workloads.