Neuron Abandoning Attention Flow: Visual Explanation of Dynamics inside CNN Models

TL;DR

This paper introduces NAFlow, a novel method for visually explaining the attention dynamics inside CNNs by tracing neuron contributions through a cascading back-propagation approach, enhancing interpretability across various tasks.

Contribution

The paper proposes a new cascading neuron abandoning back-propagation algorithm and a channel contribution weights module for visualizing attention flow in CNNs, including similarity metric-based models.

Findings

Effective in visualizing attention flow across nine CNN models

Applicable to multiple tasks including classification and image retrieval

Improves interpretability of CNN decision processes

Abstract

In this paper, we present a Neuron Abandoning Attention Flow (NAFlow) method to address the open problem of visually explaining the attention evolution dynamics inside CNNs when making their classification decisions. A novel cascading neuron abandoning back-propagation algorithm is designed to trace neurons in all layers of a CNN that involve in making its prediction to address the problem of significant interference from abandoned neurons. Firstly, a Neuron Abandoning Back-Propagation (NA-BP) module is proposed to generate Back-Propagated Feature Maps (BPFM) by using the inverse function of the intermediate layers of CNN models, on which the neurons not used for decision-making are abandoned. Meanwhile, the cascading NA-BP modules calculate the tensors of importance coefficients which are linearly combined with the tensors of BPFMs to form the NAFlow. Secondly, to be able to visualize attention flow for similarity metric-based CNN models, a new channel contribution weights module is proposed to calculate the importance coefficients via Jacobian Matrix. The effectiveness of the proposed NAFlow is validated on nine widely-used CNN models for various tasks of general image classification, contrastive learning classification, few-shot image classification, and image retrieval.