Multi-Source Knowledge Pruning for Retrieval-Augmented Generation: A Benchmark and Empirical Study

TL;DR

This paper introduces a benchmark dataset and a pruning-based framework for retrieval-augmented generation that effectively integrates multiple knowledge sources, reducing hallucinations and improving performance.

Contribution

It provides the first standardized multi-source knowledge dataset and a novel PruningRAG framework with multi-granularity pruning strategies for better knowledge integration in RAG.

Findings

PruningRAG improves performance across various RAG models.

The dataset enables comprehensive evaluation of multi-source knowledge integration.

Pruning strategies effectively reduce misleading information.

Abstract

Retrieval-augmented generation (RAG) is increasingly recognized as an effective approach to mitigating the hallucination of large language models (LLMs) through the integration of external knowledge. While numerous efforts, most studies focus on a single type of external knowledge source. However, in real-world applications, most situations involve diverse knowledge from various sources, yet this area has been less explored. The main dilemma is the lack of a suitable dataset containing multiple knowledge sources and pre-exploration of the associated issues. To address these challenges, we standardize a benchmark dataset that combines structured and unstructured knowledge across diverse and complementary domains. Based on this dataset, we further develop a plug-and-play RAG framework, \textbf{PruningRAG}, whose main characteristic is the use of multi-granularity pruning strategies to optimize the integration of relevant information while minimizing misleading context. It consistently improves performance across various existing RAG variants, demonstrating its robustness and broad applicability. Building upon the standardized dataset and PruningRAG, we also report a series of experimental results, as well as insightful findings. Our dataset and code are publicly available\footnote{https://github.com/USTCAGI/PruningRAG}, with the aim of advancing future research in the RAG community.