Separate the Wheat from the Chaff: Winnowing Down Divergent Views in Retrieval Augmented Generation

TL;DR

WinnowRAG is a novel retrieval-augmented generation framework that filters out noisy documents through a two-stage process involving clustering and iterative winnowing, significantly improving response accuracy without model fine-tuning.

Contribution

The paper introduces WinnowRAG, a model-agnostic, two-stage filtering approach that enhances RAG by systematically removing irrelevant documents to improve answer quality.

Findings

WinnowRAG outperforms state-of-the-art baselines on multiple datasets.

The framework effectively filters noise without model fine-tuning.

WinnowRAG improves the relevance and accuracy of generated responses.

Abstract

Retrieval-augmented generation (RAG) enhances large language models (LLMs) by integrating external knowledge sources to address their limitations in accessing up-to-date or specialized information. A natural strategy to increase the likelihood of retrieving relevant information is to expand the number of retrieved documents. However, involving more documents could introduce significant noise, as many documents may be irrelevant or misleading, thereby reducing the overall accuracy of the generated responses. To overcome the challenge associated with handling a larger number of documents, we propose WinnowRAG, a novel RAG framework designed to systematically filter out noisy documents while preserving valuable content -- a process we refer to as winnowing. WinnowRAG operates in two stages: In Stage I, we perform query-aware clustering to group similar documents and form distinct topic clusters. Each cluster is assigned to an LLM agent for generating a unique answer. In Stage II, we perform winnowing, wherein a critic LLM evaluates the outputs of multiple agents and iteratively separates useful documents from noisy ones. To retain useful documents when discarding agents, we propose two strategic merging techniques to ensure that only relevant knowledge is used for generating the final response. Crucially, WinnowRAG is model-agnostic and does not require any model fine-tuning, making it easily adaptable to various tasks. Extensive experiments on various realistic datasets demonstrate the effectiveness of WinnowRAG over state-of-the-art baselines.