Explaining and Interpreting LSTMs

TL;DR

This paper adapts the Layer-wise Relevance Propagation technique to LSTM networks, providing a new method for explaining their predictions by accounting for LSTM-specific structures and interactions.

Contribution

It introduces a novel propagation scheme and theoretical extension to explain LSTM predictions, bridging a gap in interpretability methods for sequential models.

Findings

Effective LRP extension for LSTMs

Faithful explanations of LSTM predictions

Enhanced interpretability of sequential models

Abstract

While neural networks have acted as a strong unifying force in the design of modern AI systems, the neural network architectures themselves remain highly heterogeneous due to the variety of tasks to be solved. In this chapter, we explore how to adapt the Layer-wise Relevance Propagation (LRP) technique used for explaining the predictions of feed-forward networks to the LSTM architecture used for sequential data modeling and forecasting. The special accumulators and gated interactions present in the LSTM require both a new propagation scheme and an extension of the underlying theoretical framework to deliver faithful explanations.