Rule-Based Explanations for Retrieval-Augmented LLM Systems

TL;DR

This paper introduces a method to generate rule-based explanations for retrieval-augmented large language models, linking retrieved sources to model outputs to improve interpretability.

Contribution

It presents the first approach to applying rule-based explanations to RAG LLMs, with optimized algorithms inspired by frequent itemset mining for efficient rule generation.

Findings

Rules effectively explain LLM outputs based on retrieved sources

Proposed algorithms significantly reduce rule generation time

Experiments demonstrate the approach's interpretability and efficiency

Abstract

If-then rules are widely used to explain machine learning models; e.g., "if employed = no, then loan application = rejected." We present the first proposal to apply rules to explain the emerging class of large language models (LLMs) with retrieval-augmented generation (RAG). Since RAG enables LLM systems to incorporate retrieved information sources at inference time, rules linking the presence or absence of sources can explain output provenance; e.g., "if a Times Higher Education ranking article is retrieved, then the LLM ranks Oxford first." To generate such rules, a brute force approach would probe the LLM with all source combinations and check if the presence or absence of any sources leads to the same output. We propose optimizations to speed up rule generation, inspired by Apriori-like pruning from frequent itemset mining but redefined within the scope of our novel problem. We conclude with qualitative and quantitative experiments demonstrating our solutions' value and efficiency.