Context-Aware Search and Retrieval Under Token Erasure

TL;DR

This paper analyzes a token erasure-aware retrieval model for RAG systems, demonstrating how importance-aware redundancy improves retrieval reliability through information-theoretic and empirical evaluations.

Contribution

It introduces a novel importance-aware redundancy scheme for token erasures in retrieval systems, supported by theoretical analysis and real-world data validation.

Findings

Higher redundancy for important features reduces retrieval error.

Similarity margins follow a multivariate Gaussian distribution asymptotically.

Importance-aware redundancy extends effectively to modern embedding-based retrieval.

Abstract

This paper introduces and analyzes a search and retrieval model for RAG-like systems under {token} erasures. We provide an information-theoretic analysis of remote document retrieval when query representations are only partially preserved. The query is represented using term-frequency-based features, and semantically adaptive redundancy is assigned according to feature importance. Retrieval is performed using TF-IDF-weighted similarity. We characterize the retrieval error probability by showing that the vector of similarity margins converges to a multivariate Gaussian distribution, yielding an explicit approximation and computable upper bounds. Numerical results support the analysis, while a separate data-driven evaluation using embedding-based retrieval on real-world data shows that the same importance-aware redundancy principles extend to modern retrieval pipelines. Overall, the results show that assigning higher redundancy to semantically important query features improves retrieval reliability.