A Note on k-NN Gating in RAG

TL;DR

This paper introduces a statistical framework for retrieval-augmented generation (RAG), formalizing how language models should optimally balance their own predictions with retrieved evidence, improving factual accuracy.

Contribution

It develops a Bayes-optimal gating mechanism for RAG, analyzes its impact on hallucination, and models distribution mismatch effects, providing a rigorous foundation for factuality in RAG systems.

Findings

Derived the Bayes-optimal per-query gate for RAG.

Analyzed the effect of gating on hallucination and factuality.

Introduced a hybrid model to handle distribution mismatch.

Abstract

We develop a statistical proxy framework for retrieval-augmented generation (RAG), designed to formalize how a language model (LM) should balance its own predictions with retrieved evidence. For each query x, the system combines a frozen base model q0 ($\times$ x) with a k-nearest neighbor retriever r (k ) ($\times$ x) through a measurable gate k(x). A retrieval-trust weight wfact (x) quantifies the geometric reliability of the retrieved neighborhood and penalizes retrieval in low-trust regions. We derive the Bayes-optimal per-query gate and analyze its effect on a discordance-based hallucination criterion that captures disagreements between LM predictions and retrieved evidence. We further show that this discordance admits a deterministic asymptotic limit governed solely by the structural agreement (or disagreement) between the Bayes rule and the LM. To account for distribution mismatch between queries and memory, we introduce a hybrid geometric-semantic model combining covariate deformation and label corruption. Overall, this note provides a principled statistical foundation for factuality-oriented RAG systems.