When saliency goes off on a tangent: Interpreting Deep Neural Networks with nonlinear saliency maps

TL;DR

This paper introduces nonlinear saliency maps that better interpret deep neural networks by capturing their nonlinearity, improving understanding of complex decision processes over traditional linear methods.

Contribution

The paper proposes a novel class of nonlinear saliency map methods that fully account for neural network nonlinearity, enhancing interpretability in complex models.

Findings

Nonlinear saliency maps outperform linear ones on complex problems

They identify more specific decision drivers in neural networks

They improve interpretability of deep learning models

Abstract

A fundamental bottleneck in utilising complex machine learning systems for critical applications has been not knowing why they do and what they do, thus preventing the development of any crucial safety protocols. To date, no method exist that can provide full insight into the granularity of the neural network's decision process. In the past, saliency maps were an early attempt at resolving this problem through sensitivity calculations, whereby dimensions of a data point are selected based on how sensitive the output of the system is to them. However, the success of saliency maps has been at best limited, mainly due to the fact that they interpret the underlying learning system through a linear approximation. We present a novel class of methods for generating nonlinear saliency maps which fully account for the nonlinearity of the underlying learning system. While agreeing with linear saliency maps on simple problems where linear saliency maps are correct, they clearly identify more specific drivers of classification on complex examples where nonlinearities are more pronounced. This new class of methods significantly aids interpretability of deep neural networks and related machine learning systems. Crucially, they provide a starting point for their more broad use in serious applications, where 'why' is equally important as 'what'.