Explaining Explanations: Axiomatic Feature Interactions for Deep Networks

TL;DR

This paper introduces Integrated Hessians, a novel method extending Integrated Gradients to explain pairwise feature interactions in neural networks, providing more comprehensive insights into model behavior.

Contribution

The paper presents Integrated Hessians, a new, architecture-agnostic method for explaining feature interactions in neural networks, addressing limitations of previous approaches.

Findings

Overcomes theoretical limitations of prior interaction explanations

Faster than existing methods with many features

Outperforms previous methods on quantitative benchmarks

Abstract

Recent work has shown great promise in explaining neural network behavior. In particular, feature attribution methods explain which features were most important to a model's prediction on a given input. However, for many tasks, simply knowing which features were important to a model's prediction may not provide enough insight to understand model behavior. The interactions between features within the model may better help us understand not only the model, but also why certain features are more important than others. In this work, we present Integrated Hessians, an extension of Integrated Gradients that explains pairwise feature interactions in neural networks. Integrated Hessians overcomes several theoretical limitations of previous methods to explain interactions, and unlike such previous methods is not limited to a specific architecture or class of neural network. Additionally, we find that our method is faster than existing methods when the number of features is large, and outperforms previous methods on existing quantitative benchmarks. Code available at https://github.com/suinleelab/path_explain