Error-margin Analysis for Hidden Neuron Activation Labels

TL;DR

This paper introduces the concept of neuron label error margin, emphasizing the importance of analyzing both recall and precision in understanding neuron representations in neural networks.

Contribution

It proposes a novel framework for analyzing the error margin of neuron labels, advancing interpretability in neural network representations.

Findings

Highlights the significance of considering both recall and precision in neuron labeling

Introduces a new metric for neuron label error margin

Provides insights into neuron response variability

Abstract

Understanding how high-level concepts are represented within artificial neural networks is a fundamental challenge in the field of artificial intelligence. While existing literature in explainable AI emphasizes the importance of labeling neurons with concepts to understand their functioning, they mostly focus on identifying what stimulus activates a neuron in most cases, this corresponds to the notion of recall in information retrieval. We argue that this is only the first-part of a two-part job, it is imperative to also investigate neuron responses to other stimuli, i.e., their precision. We call this the neuron labels error margin.