Neural Response Interpretation through the Lens of Critical Pathways

TL;DR

This paper investigates how to identify and utilize critical neural pathways for interpreting network responses, proposing a neuron contribution-based pathway selection method and a new feature attribution technique called 'pathway gradient' validated through experiments.

Contribution

It introduces a neuron contribution-based pathway selection method that ensures critical input features are included, and proposes 'pathway gradient' for feature attribution, validated by experiments.

Findings

Pathways selected via neuron contribution are locally linear.

Pathway gradient effectively attributes critical input features.

Selected pathways correspond to critical input features.

Abstract

Is critical input information encoded in specific sparse pathways within the neural network? In this work, we discuss the problem of identifying these critical pathways and subsequently leverage them for interpreting the network's response to an input. The pruning objective -- selecting the smallest group of neurons for which the response remains equivalent to the original network -- has been previously proposed for identifying critical pathways. We demonstrate that sparse pathways derived from pruning do not necessarily encode critical input information. To ensure sparse pathways include critical fragments of the encoded input information, we propose pathway selection via neurons' contribution to the response. We proceed to explain how critical pathways can reveal critical input features. We prove that pathways selected via neuron contribution are locally linear (in an L2-ball), a property that we use for proposing a feature attribution method: "pathway gradient". We validate our interpretation method using mainstream evaluation experiments. The validation of pathway gradient interpretation method further confirms that selected pathways using neuron contributions correspond to critical input features. The code is publicly available.