In Search of Grandmother Cells: Tracing Interpretable Neurons in Tabular Representations

TL;DR

This paper investigates whether neurons in tabular foundation models exhibit interpretability akin to grandmother cells, using information-theoretic measures to identify neurons that respond selectively to high-level concepts.

Contribution

The study introduces two information-theoretic metrics to quantify neuron saliency and selectivity, and demonstrates their effectiveness in identifying interpretable neurons in tabular models.

Findings

Some neurons show significant saliency for high-level concepts.

Interpretable neurons can emerge naturally in foundation models.

Simple search methods can identify neurons with high interpretability.

Abstract

Foundation models are powerful yet often opaque in their decision-making. A topic of continued interest in both neuroscience and artificial intelligence is whether some neurons behave like grandmother cells, i.e., neurons that are inherently interpretable because they exclusively respond to single concepts. In this work, we propose two information-theoretic measures that quantify the neuronal saliency and selectivity for single concepts. We apply these metrics to the representations of TabPFN, a tabular foundation model, and perform a simple search across neuron-concept pairs to find the most salient and selective pair. Our analysis provides the first evidence that some neurons in such models show moderate, statistically significant saliency and selectivity for high-level concepts. These findings suggest that interpretable neurons can emerge naturally and that they can, in some cases, be identified without resorting to more complex interpretability techniques.