NeuroInspect: Interpretable Neuron-based Debugging Framework through Class-conditional Visualizations

TL;DR

NeuroInspect is a neuron-based debugging framework that visualizes and mitigates errors in deep learning models by pinpointing responsible neurons and providing human-interpretable explanations without needing extra data.

Contribution

It introduces CLIP-Illusion for class-conditional feature visualization and a method to mitigate false correlations, enhancing interpretability and debugging of neural networks.

Findings

Effective identification of neurons responsible for errors.

Improved interpretability through class-conditional visualizations.

Reduction of false correlations in model decisions.

Abstract

Despite deep learning (DL) has achieved remarkable progress in various domains, the DL models are still prone to making mistakes. This issue necessitates effective debugging tools for DL practitioners to interpret the decision-making process within the networks. However, existing debugging methods often demand extra data or adjustments to the decision process, limiting their applicability. To tackle this problem, we present NeuroInspect, an interpretable neuron-based debugging framework with three key stages: counterfactual explanations, feature visualizations, and false correlation mitigation. Our debugging framework first pinpoints neurons responsible for mistakes in the network and then visualizes features embedded in the neurons to be human-interpretable. To provide these explanations, we introduce CLIP-Illusion, a novel feature visualization method that generates images representing features conditioned on classes to examine the connection between neurons and the decision layer. We alleviate convoluted explanations of the conventional visualization approach by employing class information, thereby isolating mixed properties. This process offers more human-interpretable explanations for model errors without altering the trained network or requiring additional data. Furthermore, our framework mitigates false correlations learned from a dataset under a stochastic perspective, modifying decisions for the neurons considered as the main causes. We validate the effectiveness of our framework by addressing false correlations and improving inferences for classes with the worst performance in real-world settings. Moreover, we demonstrate that NeuroInspect helps debug the mistakes of DL models through evaluation for human understanding. The code is openly available at https://github.com/yeongjoonJu/NeuroInspect.