REAR: A Relevance-Aware Retrieval-Augmented Framework for Open-Domain Question Answering

TL;DR

REAR introduces a relevance-aware framework for open-domain question answering that improves the utilization of external knowledge by assessing document relevance, leading to significant performance gains over previous RAG methods.

Contribution

The paper proposes a novel architecture and training method for RAG systems that enhances relevance assessment, improving external knowledge utilization in open-domain QA.

Findings

REAR outperforms previous RAG approaches on four QA tasks.

The relevance assessment module improves the accuracy of external knowledge usage.

Enhanced training methods increase the robustness and effectiveness of the model.

Abstract

Considering the limited internal parametric knowledge, retrieval-augmented generation (RAG) has been widely used to extend the knowledge scope of large language models (LLMs). Despite the extensive efforts on RAG research, in existing methods, LLMs cannot precisely assess the relevance of retrieved documents, thus likely leading to misleading or even incorrect utilization of external knowledge (eg., retrieved documents). To address this issue, in this paper, we propose REAR, a RElevance-Aware Retrieval-augmented approach for open-domain question answering (QA). As the key motivation, we aim to enhance the self-awareness regarding the reliability of external knowledge for LLMs, so as to adaptively utilize external knowledge in RAG systems. Specially, we develop a novel architecture for LLM-based RAG systems, by incorporating a specially designed assessment module that precisely assesses the relevance of retrieved documents. Furthermore, we propose an improved training method based on bi-granularity relevance fusion and noise-resistant training. By combining the improvements in both architecture and training, our proposed REAR can better utilize external knowledge by effectively perceiving the relevance of retrieved documents. Experiments on four open-domain QA tasks show that REAR significantly outperforms previous a number of competitive RAG approaches. Our codes can be accessed at https://github.com/RUCAIBox/REAR.