CORE-RAG: Lossless Compression for Retrieval-Augmented LLMs via Reinforcement Learning

TL;DR

CORE-RAG introduces a lossless, reinforcement learning-based compression method for retrieval-augmented large language models, significantly reducing input size while maintaining or improving task performance.

Contribution

It proposes a novel end-to-end optimized lossless compression technique that uses downstream task feedback, avoiding predefined heuristics and enhancing RAG efficiency.

Findings

Achieves a 3% compression ratio without performance loss.

Improves average Exact Match score by 3.3 points.

Demonstrates effectiveness across four datasets.

Abstract

Retrieval-Augmented Generation (RAG) has emerged as a promising approach to enhance the timeliness of knowledge updates and the factual accuracy of responses in large language models. However, incorporating a large number of retrieved documents significantly increases input length, leading to higher computational costs. Existing approaches to document compression tailored for RAG often degrade task performance, as they typically rely on predefined heuristics in the absence of clear compression guidelines. These heuristics fail to ensure that the compressed content effectively supports downstream tasks. To address these limitations, we propose CORE, a novel method for lossless context compression in RAG. CORE is optimized end-to-end and does not depend on predefined compression labels, which are often impractical to obtain. Instead, it leverages downstream task performance as a feedback signal, iteratively refining the compression policy to enhance task effectiveness. Extensive experiments across four datasets demonstrate the effectiveness of CORE. With a high compression ratio of 3%, CORE not only prevents performance degradation compared to including full documents (i.e., without compression) but also improves the average Exact Match (EM) score by 3.3 points. The code for CORE will be released soon.