Select, Hypothesize and Verify: Towards Verified Neuron Concept Interpretation

TL;DR

This paper introduces a framework for interpreting neuron functions in neural networks by selecting activation samples, hypothesizing concepts, and verifying their accuracy, leading to more reliable neuron concept descriptions.

Contribution

It proposes a novel Select-Hypothesize-Verify framework that improves the accuracy of neuron concept interpretation by verifying the relevance of generated concepts.

Findings

Generated concepts activate neurons 1.5 times more accurately than previous methods.

The framework effectively filters out misleading neuron descriptions.

Experiments demonstrate enhanced interpretability of neuron functions.

Abstract

It is essential for understanding neural network decisions to interpret the functionality (also known as concepts) of neurons. Existing approaches describe neuron concepts by generating natural language descriptions, thereby advancing the understanding of the neural network's decision-making mechanism. However, these approaches assume that each neuron has well-defined functions and provides discriminative features for neural network decision-making. In fact, some neurons may be redundant or may offer misleading concepts. Thus, the descriptions for such neurons may cause misinterpretations of the factors driving the neural network's decisions. To address the issue, we introduce a verification of neuron functions, which checks whether the generated concept highly activates the corresponding neuron. Furthermore, we propose a Select-Hypothesize-Verify framework for interpreting neuron functionality. This framework consists of: 1) selecting activation samples that best capture a neuron's well-defined functional behavior through activation-distribution analysis; 2) forming hypotheses about concepts for the selected neurons; and 3) verifying whether the generated concepts accurately reflect the functionality of the neuron. Extensive experiments show that our method produces more accurate neuron concepts. Our generated concepts activate the corresponding neurons with a probability approximately 1.5 times that of the current state-of-the-art method.