Predictive Prefetching for Retrieval-Augmented Generation

TL;DR

This paper introduces an advanced asynchronous retrieval framework with predictive prefetching for RAG models, significantly reducing latency while maintaining answer quality.

Contribution

It proposes a novel framework that predicts retrieval needs during generation, improving efficiency in complex, multi-domain settings.

Findings

Up to 43.5% latency reduction

62.4% improvement in time-to-first-token

Maintains answer quality comparable to baselines

Abstract

Retrieval-Augmented Generation (RAG) improves factual grounding in large language models but suffers from substantial latency due to synchronous retrieval. While recent work explores asynchronous retrieval, existing approaches rely on heuristic coordination between retrieval and generation and assume stable information demands during decoding that often break in complex, multi-domain settings. In this paper, we propose an advanced asynchronous retrieval framework that enables predictive prefetching aligned with evolving information needs. The framework explicitly predicts when retrieval should be triggered and what information should be retrieved using three components, a retrieval predictor, a context monitor, and a query generator, by exploiting semantic precursors in generation dynamics that emerge several tokens before uncertainty becomes critical. Experiments on multiple benchmarks demonstrate up to 43.5% end-to-end latency reduction and 62.4% improvement in time-to-first-token, while maintaining answer quality comparable to synchronous RAG baselines.