RuleRAG: Rule-Guided Retrieval-Augmented Generation with Language Models for Question Answering

TL;DR

RuleRAG introduces rule-guided retrieval and reasoning in retrieval-augmented generation for question answering, significantly improving retrieval accuracy and answer correctness by leveraging rules and knowledge graphs.

Contribution

It proposes a novel rule-guided framework (RuleRAG) with explicit rules for retrieval and reasoning, and constructs rule-aware benchmarks based on knowledge graphs.

Findings

Retrieval recall improved by 89.2% with RuleRAG-ICL.

Answer accuracy increased by 103.1% with RuleRAG-ICL.

Effective generalization of rule guidance across datasets.

Abstract

Retrieval-augmented generation (RAG) has shown promising potential in knowledge intensive question answering (QA). However, existing approaches only consider the query itself, neither specifying the retrieval preferences for the retrievers nor informing the generators of how to refer to the retrieved documents for the answers, which poses a significant challenge to the QA performance. To address these issues, we propose Rule-guided Retrieval-Augmented Generation with LMs, which explicitly introduces rules for in-context learning (RuleRAG-ICL) to guide retrievers to recall related documents in the directions of rules and uniformly guide generators to reason attributed by the same rules. Moreover, most existing RAG datasets were constructed without considering rules and Knowledge Graphs (KGs) are recognized as providing high-quality rules. Therefore, we construct five rule-aware RAG benchmarks for QA, RuleQA, based on KGs to stress the significance of retrieval and reasoning with rules. Experiments on RuleQA demonstrate RuleRAG-ICL improves the retrieval quality of +89.2% in Recall@10 and answer accuracy of +103.1% in Exact Match, and RuleRAG-FT yields more enhancement. In addition, experiments on four existing RAG datasets show RuleRAG is also effective by offering rules in RuleQA to them, further proving the generalization of rule guidance in RuleRAG.