Don't Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Tasks

TL;DR

This paper introduces cache-augmented generation (CAG), an approach that leverages large language models' extended context to bypass retrieval, reducing latency and errors while maintaining performance in knowledge tasks.

Contribution

The paper proposes CAG as a retrieval-free alternative to RAG, utilizing preloaded knowledge in LLMs' context to improve efficiency and simplicity for certain applications.

Findings

CAG eliminates retrieval latency and errors.

CAG achieves comparable or better performance than RAG.

CAG is effective when knowledge base is limited.

Abstract

Retrieval-augmented generation (RAG) has gained traction as a powerful approach for enhancing language models by integrating external knowledge sources. However, RAG introduces challenges such as retrieval latency, potential errors in document selection, and increased system complexity. With the advent of large language models (LLMs) featuring significantly extended context windows, this paper proposes an alternative paradigm, cache-augmented generation (CAG) that bypasses real-time retrieval. Our method involves preloading all relevant resources, especially when the documents or knowledge for retrieval are of a limited and manageable size, into the LLM's extended context and caching its runtime parameters. During inference, the model utilizes these preloaded parameters to answer queries without additional retrieval steps. Comparative analyses reveal that CAG eliminates retrieval latency and minimizes retrieval errors while maintaining context relevance. Performance evaluations across multiple benchmarks highlight scenarios where long-context LLMs either outperform or complement traditional RAG pipelines. These findings suggest that, for certain applications, particularly those with a constrained knowledge base, CAG provide a streamlined and efficient alternative to RAG, achieving comparable or superior results with reduced complexity.