Causal Interpretation of Neural Network Computations with Contribution Decomposition

TL;DR

This paper introduces CODEC, a method that decomposes neural network behavior into sparse, causal contribution modes, enabling better interpretation, control, and understanding of hierarchical nonlinear computations in both artificial and biological neural systems.

Contribution

We propose CODEC, a novel contribution decomposition technique using autoencoders to reveal causal, sparse motifs of neuron contributions in neural networks and biological models.

Findings

Contributions become sparser and more dimensional across layers.

Positive and negative effects on outputs tend to decorrelate in deeper layers.

Decomposition enables targeted manipulation and visualization of network behavior.

Abstract

Understanding how neural networks transform inputs into outputs is crucial for interpreting and manipulating their behavior. Most existing approaches analyze internal representations by identifying hidden-layer activation patterns correlated with human-interpretable concepts. Here we take a direct approach to examine how hidden neurons act to drive network outputs. We introduce CODEC (Contribution Decomposition), a method that uses sparse autoencoders to decompose network behavior into sparse motifs of hidden-neuron contributions, revealing causal processes that cannot be determined by analyzing activations alone. Applying CODEC to benchmark image-classification networks, we find that contributions grow in sparsity and dimensionality across layers and, unexpectedly, that they progressively decorrelate positive and negative effects on network outputs. We further show that decomposing contributions into sparse modes enables greater control and interpretation of intermediate layers, supporting both causal manipulations of network output and human-interpretable visualizations of distinct image components that combine to drive that output. Finally, by analyzing state-of-the-art models of neural activity in the vertebrate retina, we demonstrate that CODEC uncovers combinatorial actions of model interneurons and identifies the sources of dynamic receptive fields. Overall, CODEC provides a rich and interpretable framework for understanding how nonlinear computations evolve across hierarchical layers, establishing contribution modes as an informative unit of analysis for mechanistic insights into artificial neural networks.