AdaComp: Extractive Context Compression with Adaptive Predictor for Retrieval-Augmented Large Language Models

TL;DR

AdaComp is a novel adaptive context compression method for retrieval-augmented large language models that reduces inference costs by predicting the optimal number of documents needed per query.

Contribution

It introduces a low-cost, adaptive compression technique that dynamically determines the number of documents to retain based on query complexity and retrieval quality.

Findings

Significantly reduces inference costs in RAG systems.

Maintains near-original performance levels with compressed context.

Effective across multiple QA datasets and conversational settings.

Abstract

Retrieved documents containing noise will hinder RAG from detecting answer clues and make the inference process slow and expensive. Therefore, context compression is necessary to enhance its accuracy and efficiency. Existing context compression methods use extractive or generative models to retain the most query-relevant sentences or apply the information bottleneck theory to preserve sufficient information. However, these methods may face issues such as over-compression or high computational costs. We observe that the retriever often ranks relevant documents at the top, but the exact number of documents needed to answer the query is uncertain due to the impact of query complexity and retrieval quality: complex queries like multi-hop questions may require retaining more documents than simpler queries, and a low-quality retrieval may need to rely on more documents to generate accurate outputs. Therefore, determining the minimum number of required documents (compression rate) is still a challenge for RAG. In this paper, we introduce AdaComp, a low-cost extractive context compression method that adaptively determines the compression rate based on both query complexity and retrieval quality. Specifically, we first annotate the minimum top-k documents necessary for the RAG system to answer the current query as the compression rate and then construct triplets of the query, retrieved documents, and its compression rate. Then, we use this triplet dataset to train a compression-rate predictor. Experiments on three QA datasets and one conversational Multi-doc QA dataset show that AdaComp significantly reduces inference costs while maintaining performance nearly identical to uncompressed models, achieving a balance between efficiency and performance.