Noise-Robust Abstractive Compression in Retrieval-Augmented Language Models

TL;DR

This paper introduces ACoRN, a noise-robust abstractive compression method for retrieval-augmented models, improving answer accuracy by handling irrelevant or misleading retrieved documents through novel training strategies.

Contribution

We propose ACoRN, a new training framework that enhances the robustness of language model compressors against retrieval noise, improving factual accuracy in long-context summarization.

Findings

ACoRN improves EM and F1 scores on relevant datasets.

ACoRN maintains answer string fidelity, providing direct evidence.

The method is especially effective with noisy retrieval scenarios.

Abstract

Abstractive compression utilizes smaller langauge models to condense query-relevant context, reducing computational costs in retrieval-augmented generation (RAG). However, retrieved documents often include information that is either irrelevant to answering the query or misleading due to factual incorrect content, despite having high relevance scores. This behavior indicates that abstractive compressors are more likely to omit important information essential for the correct answer, especially in long contexts where attention dispersion occurs. To address this issue, we categorize retrieved documents in a more fine-grained manner and propose Abstractive Compression Robust against Noise (ACoRN), which introduces two novel training steps. First, we use offline data augmentation on the training dataset to enhance compressor robustness against two distinct types of retrieval noise. Second, since the language model based compressor cannot fully utilize information from multiple retrieved documents and exhibits positional bias, we perform finetuning to generate summaries centered around key information that directly supports the correct answer. Our experiments demonstrate that T5-large, trained with ACoRN as a compressor, improves EM and F1 scores while preserving the answer string, which could serve as direct evidence. ACoRN excels on datasets with many accuracy reducing documents, making it highly useful in real-world scenarios.