Quantifying Document Impact in RAG-LLMs

TL;DR

This paper introduces the Influence Score (IS), a new metric based on Partial Information Decomposition, to quantify the impact of individual documents on RAG-generated responses, enhancing transparency and trustworthiness.

Contribution

The paper proposes the Influence Score (IS), a novel metric for measuring document impact in RAG systems, validated through experiments demonstrating its effectiveness in identifying influential documents.

Findings

IS correctly identifies malicious documents in 86% of cases

Using top-ranked documents by IS yields responses closer to original

IS improves transparency and reliability of RAG systems

Abstract

Retrieval Augmented Generation (RAG) enhances Large Language Models (LLMs) by connecting them to external knowledge, improving accuracy and reducing outdated information. However, this introduces challenges such as factual inconsistencies, source conflicts, bias propagation, and security vulnerabilities, which undermine the trustworthiness of RAG systems. A key gap in current RAG evaluation is the lack of a metric to quantify the contribution of individual retrieved documents to the final output. To address this, we introduce the Influence Score (IS), a novel metric based on Partial Information Decomposition that measures the impact of each retrieved document on the generated response. We validate IS through two experiments. First, a poison attack simulation across three datasets demonstrates that IS correctly identifies the malicious document as the most influential in $86\%$ of cases. Second, an ablation study shows that a response generated using only the top-ranked documents by IS is consistently judged more similar to the original response than one generated from the remaining documents. These results confirm the efficacy of IS in isolating and quantifying document influence, offering a valuable tool for improving the transparency and reliability of RAG systems.