Retrieval-Augmented Generation as Noisy In-Context Learning: A Unified Theory and Risk Bounds

TL;DR

This paper provides the first theoretical generalization bounds for retrieval-augmented generation (RAG), modeling it as noisy in-context learning and analyzing its bias-variance tradeoff, with empirical validation on QA benchmarks.

Contribution

It introduces a finite-sample generalization bound for RAG, unifies RAG and in-context learning under a common framework, and models retrieval noise from various sources.

Findings

RAG has an intrinsic ceiling on generalization error.

Sample efficiency of RAG and ICL is empirically validated.

Theoretical analysis reveals bias-variance tradeoff in RAG.

Abstract

Retrieval-augmented generation (RAG) has seen many empirical successes in recent years by aiding the LLM with external knowledge. However, its theoretical aspect has remained mostly unexplored. In this paper, we propose the first finite-sample generalization bound for RAG in in-context linear regression and derive an exact bias-variance tradeoff. Our framework views the retrieved texts as query-dependent noisy in-context examples and recovers the classical in-context learning (ICL) and standard RAG as the limit cases. Our analysis suggests that an intrinsic ceiling on generalization error exists on RAG as opposed to the ICL. Furthermore, our framework is able to model retrieval both from the training data and from external corpora by introducing uniform and non-uniform RAG noise. In line with our theory, we show the sample efficiency of ICL and RAG empirically with experiments on common QA benchmarks, such as Natural Questions and TriviaQA.