Explaining Local, Global, And Higher-Order Interactions In Deep Learning

TL;DR

This paper introduces a generalizable method using cross derivatives to explain feature interactions in neural networks, extending existing tools like Grad-CAM to higher-order interactions and demonstrating its effectiveness in vision tasks.

Contribution

The paper presents a novel cross derivative-based algorithm for explaining multi-way feature interactions and extends Grad-CAM to higher-order interactions, improving interpretability in neural networks.

Findings

Cross derivatives outperform weight-based attribution in detecting interactions.

Taylor-CAM effectively explains relational reasoning in images.

The method is validated through qualitative, quantitative, and user study results.

Abstract

We present a simple yet highly generalizable method for explaining interacting parts within a neural network's reasoning process. First, we design an algorithm based on cross derivatives for computing statistical interaction effects between individual features, which is generalized to both 2-way and higher-order (3-way or more) interactions. We present results side by side with a weight-based attribution technique, corroborating that cross derivatives are a superior metric for both 2-way and higher-order interaction detection. Moreover, we extend the use of cross derivatives as an explanatory device in neural networks to the computer vision setting by expanding Grad-CAM, a popular gradient-based explanatory tool for CNNs, to the higher order. While Grad-CAM can only explain the importance of individual objects in images, our method, which we call Taylor-CAM, can explain a neural network's relational reasoning across multiple objects. We show the success of our explanations both qualitatively and quantitatively, including with a user study. We will release all code as a tool package to facilitate explainable deep learning.