Pathwise Explanation of ReLU Neural Networks

TL;DR

This paper introduces a pathwise explanation method for ReLU neural networks that enhances interpretability by analyzing decision paths, offering flexible and detailed explanations that outperform existing methods.

Contribution

It presents a novel pathwise explanation approach that considers subsets of hidden units, improving transparency and interpretability of ReLU networks.

Findings

Outperforms existing explanation methods quantitatively.

Provides more consistent and detailed explanations.

Flexible adjustment of explanation scope within inputs.

Abstract

Neural networks have demonstrated a wide range of successes, but their ``black box" nature raises concerns about transparency and reliability. Previous research on ReLU networks has sought to unwrap these networks into linear models based on activation states of all hidden units. In this paper, we introduce a novel approach that considers subsets of the hidden units involved in the decision making path. This pathwise explanation provides a clearer and more consistent understanding of the relationship between the input and the decision-making process. Our method also offers flexibility in adjusting the range of explanations within the input, i.e., from an overall attribution input to particular components within the input. Furthermore, it allows for the decomposition of explanations for a given input for more detailed explanations. Experiments demonstrate that our method outperforms others both quantitatively and qualitatively.