CaGR-RAG: Context-aware Query Grouping for Disk-based Vector Search in RAG Systems

TL;DR

CaGR-RAG is a novel query grouping method for disk-based vector search in RAG systems that improves cache efficiency and reduces latency by organizing queries based on shared cluster access patterns.

Contribution

It introduces a context-aware query grouping and prefetching mechanism that optimizes disk access patterns in vector search systems.

Findings

Reduces 99th percentile tail latency by up to 51.55%.

Maintains higher cache hit ratio than baseline methods.

Improves overall retrieval performance in RAG systems.

Abstract

Modern embedding models capture both semantic and syntactic structures of queries, often mapping different queries to similar regions in vector space. This results in non-uniform cluster access patterns in disk-based vector search systems, particularly in Retrieval Augmented Generation (RAG) framework. While existing approaches optimize individual queries, they overlook the impact of cluster access patterns, failing to account for the locality effects of queries that access similar clusters. This oversight reduces cache efficiency and increases search latency due to excessive disk I/O. To address this, we introduce CaGR-RAG, a context-aware query grouping mechanism that organizes queries based on shared cluster access patterns. Additionally, it incorporates opportunistic cluster prefetching to minimize cache misses during transitions between query groups, further optimizing retrieval performance. Experimental results show that CaGR-RAG reduces 99th percentile tail latency by up to 51.55% while consistently maintaining a higher cache hit ratio than the baseline.