Fantasy: Efficient Large-scale Vector Search on GPU Clusters with GPUDirect Async

TL;DR

Fantasy is a GPU cluster system that enhances large-scale vector search efficiency by overlapping computation and data transfer using GPUDirect Async, enabling high throughput for AI applications like LLMs.

Contribution

It introduces Fantasy, a novel system that pipelines vector search and data transfer in GPU clusters, overcoming memory limitations and reducing stalls in large-scale vector similarity search.

Findings

Significantly improves search throughput for large graphs

Enables large query batch processing with reduced latency

Efficiently overlaps computation and communication in GPU clusters

Abstract

Vector similarity search has become a critical component in AI-driven applications such as large language models (LLMs). To achieve high recall and low latency, GPUs are utilized to exploit massive parallelism for faster query processing. However, as the number of vectors continues to grow, the graph size quickly exceeds the memory capacity of a single GPU, making it infeasible to store and process the entire index on a single GPU. Recent work uses CPU-GPU architectures to keep vectors in CPU memory or SSDs, but the loading step stalls GPU computation. We present Fantasy, an efficient system that pipelines vector search and data transfer in a GPU cluster with GPUDirect Async. Fantasy overlaps computation and network communication to significantly improve search throughput for large graphs and deliver large query batch sizes.