ReFilter: Improving Robustness of Retrieval-Augmented Generation via Gated Filter

TL;DR

ReFilter introduces a token-level filtering and fusion method for retrieval-augmented generation, enhancing robustness and performance in knowledge-intensive QA tasks, especially with large retrieval sets.

Contribution

The paper proposes ReFilter, a novel latent-based fusion framework with token-level filtering and gating, improving scalability and effectiveness over existing methods in RAG systems.

Findings

ReFilter outperforms existing fusion methods on four general-domain QA benchmarks.

ReFilter achieves 70.01% accuracy on biomedical QA benchmarks without domain fine-tuning.

The approach scales well with larger retrieval sets, maintaining high performance.

Abstract

Retrieval-augmented generation (RAG) has become a dominant paradigm for grounding large language models (LLMs) with external evidence in knowledge-intensive question answering. A core design choice is how to fuse retrieved samples into the LLMs, where existing internal fusion approaches broadly fall into query-based fusion, parametric fusion, and latent-based fusion. Despite their effectiveness at modest retrieval scales, these methods often fail to scale gracefully as the number of retrieved candidates k increases: Larger k improves evidence coverage, yet realistic top-k retrieval inevitably contains irrelevant or redundant content and increases the inference cost. To address these limitations, we propose ReFilter, a novel latent-based fusion framework that performs token-level filtering and fusion. ReFilter consists of three key components: a context encoder for encoding context features, a gated filter for weighting each token, and a token fusion module for integrating the weighted token feature into the LLM's hidden states. Our experiments across four general-domain QA benchmarks show that ReFilter consistently achieves the best average performance under both in-domain adaptation and out-of-domain transfer. ReFilter further generalizes to five biomedical QA benchmarks in zero-shot transfer without domain fine-tuning, reaching 70.01% average accuracy with Qwen2.5-14B-Instruct.