AttriBoT: A Bag of Tricks for Efficiently Approximating Leave-One-Out Context Attribution

TL;DR

AttriBoT introduces a set of techniques that significantly accelerate leave-one-out context attribution for large language models, enabling scalable and faithful interpretability with over 300 times speedup.

Contribution

The paper presents novel methods for efficiently approximating LOO error in context attribution, combining caching, hierarchical attribution, and proxy models for large LLMs.

Findings

Achieves over 300x speedup in computing context attributions.

Provides more faithful LOO error approximation than prior methods.

Enables attributions to be computed 30x faster than generating responses.

Abstract

The influence of contextual input on the behavior of large language models (LLMs) has prompted the development of context attribution methods that aim to quantify each context span's effect on an LLM's generations. The leave-one-out (LOO) error, which measures the change in the likelihood of the LLM's response when a given span of the context is removed, provides a principled way to perform context attribution, but can be prohibitively expensive to compute for large models. In this work, we introduce AttriBoT, a series of novel techniques for efficiently computing an approximation of the LOO error for context attribution. Specifically, AttriBoT uses cached activations to avoid redundant operations, performs hierarchical attribution to reduce computation, and emulates the behavior of large target models with smaller proxy models. Taken together, AttriBoT can provide a >300x speedup while remaining more faithful to a target model's LOO error than prior context attribution methods. This stark increase in performance makes computing context attributions for a given response 30x faster than generating the response itself, empowering real-world applications that require computing attributions at scale. We release a user-friendly and efficient implementation of AttriBoT to enable efficient LLM interpretability as well as encourage future development of efficient context attribution methods.