Explaining the Reasoning of Large Language Models Using Attribution Graphs

TL;DR

This paper introduces CAGE, a novel graph-based framework that enhances the explanation of large language models' reasoning by capturing inter-generational influences, significantly improving attribution faithfulness.

Contribution

The paper proposes the CAGE framework, which constructs attribution graphs to better explain LLM reasoning by considering influences from all prior generations, addressing limitations of existing methods.

Findings

CAGE improves attribution faithfulness by up to 40%.

It effectively captures inter-generational influences in LLMs.

The method is validated across multiple models and datasets.

Abstract

Large language models (LLMs) exhibit remarkable capabilities, yet their reasoning remains opaque, raising safety and trust concerns. Attribution methods, which assign credit to input features, have proven effective for explaining the decision making of computer vision models. From these, context attributions have emerged as a promising approach for explaining the behavior of autoregressive LLMs. However, current context attributions produce incomplete explanations by directly relating generated tokens to the prompt, discarding inter-generational influence in the process. To overcome these shortcomings, we introduce the Context Attribution via Graph Explanations (CAGE) framework. CAGE introduces an attribution graph: a directed graph that quantifies how each generation is influenced by both the prompt and all prior generations. The graph is constructed to preserve two properties-causality and row stochasticity. The attribution graph allows context attributions to be computed by marginalizing intermediate contributions along paths in the graph. Across multiple models, datasets, metrics, and methods, CAGE improves context attribution faithfulness, achieving average gains of up to 40%.