An Information Bottleneck Perspective for Effective Noise Filtering on Retrieval-Augmented Generation

TL;DR

This paper introduces an information bottleneck approach to improve noise filtering in retrieval-augmented generation, enhancing answer accuracy and conciseness by optimizing mutual information measures.

Contribution

It applies information bottleneck theory to retrieval-augmented generation, providing a new framework for effective noise filtering and evaluation in large language models.

Findings

Achieves significant improvements in question answering accuracy.

Reduces noise with only 2.5% compression rate.

Enhances answer conciseness and correctness.

Abstract

Retrieval-augmented generation integrates the capabilities of large language models with relevant information retrieved from an extensive corpus, yet encounters challenges when confronted with real-world noisy data. One recent solution is to train a filter module to find relevant content but only achieve suboptimal noise compression. In this paper, we propose to introduce the information bottleneck theory into retrieval-augmented generation. Our approach involves the filtration of noise by simultaneously maximizing the mutual information between compression and ground output, while minimizing the mutual information between compression and retrieved passage. In addition, we derive the formula of information bottleneck to facilitate its application in novel comprehensive evaluations, the selection of supervised fine-tuning data, and the construction of reinforcement learning rewards. Experimental results demonstrate that our approach achieves significant improvements across various question answering datasets, not only in terms of the correctness of answer generation but also in the conciseness with $2.5\%$ compression rate.