Generalizing Backpropagation for Gradient-Based Interpretability

TL;DR

This paper extends backpropagation to compute various interpretability metrics in neural networks, enabling deeper insights into model behavior and feature importance beyond standard gradients.

Contribution

It introduces a generalized backpropagation algorithm based on semirings, allowing efficient computation of diverse interpretability statistics in neural networks.

Findings

Gradient flow correlates with feature importance.

Identified key pathways in BERT's attention for subject-verb agreement.

Validated the interpretability method on synthetic and real datasets.

Abstract

Many popular feature-attribution methods for interpreting deep neural networks rely on computing the gradients of a model's output with respect to its inputs. While these methods can indicate which input features may be important for the model's prediction, they reveal little about the inner workings of the model itself. In this paper, we observe that the gradient computation of a model is a special case of a more general formulation using semirings. This observation allows us to generalize the backpropagation algorithm to efficiently compute other interpretable statistics about the gradient graph of a neural network, such as the highest-weighted path and entropy. We implement this generalized algorithm, evaluate it on synthetic datasets to better understand the statistics it computes, and apply it to study BERT's behavior on the subject-verb number agreement task (SVA). With this method, we (a) validate that the amount of gradient flow through a component of a model reflects its importance to a prediction and (b) for SVA, identify which pathways of the self-attention mechanism are most important.