Evaluating Attribution Methods using White-Box LSTMs

TL;DR

This paper introduces a framework for evaluating neural network interpretability methods using transparent white-box LSTM models, revealing that existing attribution methods often fail to produce accurate explanations even on well-understood models.

Contribution

It proposes a novel evaluation framework using white-box networks to systematically assess attribution methods, highlighting their limitations.

Findings

All five attribution methods failed to produce expected explanations.

White-box LSTMs solved tasks perfectly and transparently.

Evaluation framework can identify shortcomings of interpretability methods.

Abstract

Interpretability methods for neural networks are difficult to evaluate because we do not understand the black-box models typically used to test them. This paper proposes a framework in which interpretability methods are evaluated using manually constructed networks, which we call white-box networks, whose behavior is understood a priori. We evaluate five methods for producing attribution heatmaps by applying them to white-box LSTM classifiers for tasks based on formal languages. Although our white-box classifiers solve their tasks perfectly and transparently, we find that all five attribution methods fail to produce the expected model explanations.