What should a neuron aim for? Designing local objective functions based on information theory
Andreas C. Schneider, Valentin Neuhaus, David A. Ehrlich, Abdullah Makkeh, Alexander S. Ecker, Viola Priesemann, Michael Wibral
TL;DR
This paper proposes a novel information-theoretic framework for designing local learning objectives in neural networks, inspired by biological self-organization, to improve interpretability and performance.
Contribution
It introduces a bio-inspired local learning goal based on Partial Information Decomposition, enabling neurons to selectively integrate input information in a task-relevant manner.
Findings
Neurons can be guided to prioritize unique, redundant, or synergistic information.
The framework enhances interpretability of neuron functions.
Potential for improved local learning in deep networks.
Abstract
In modern deep neural networks, the learning dynamics of the individual neurons is often obscure, as the networks are trained via global optimization. Conversely, biological systems build on self-organized, local learning, achieving robustness and efficiency with limited global information. We here show how self-organization between individual artificial neurons can be achieved by designing abstract bio-inspired local learning goals. These goals are parameterized using a recent extension of information theory, Partial Information Decomposition (PID), which decomposes the information that a set of information sources holds about an outcome into unique, redundant and synergistic contributions. Our framework enables neurons to locally shape the integration of information from various input classes, i.e. feedforward, feedback, and lateral, by selecting which of the three inputs should…
Peer Reviews
Decision·ICLR 2025 Oral
1. The topic of biologically inspired neural networks has attracted great attention in the recent years. Proposing alternative solutions to the long-dominating backpropagation method is also interesting and profoundly influential. I encourage the authors to work along these lines. 2. Overall this is a paper very rich in content. 3. The visualizations in Figure 1 and 2 are very helpful for understanding the partial information components in partial information decomposition. 4. The performance on
1. While I believe we should not expect super comprehensive experiments from a paper that initially introduce a novel concept or paradigm, it might be even improve the soundness of this paper if the authors can also include the performance on slightly more challenging datasets. If standard large datasets such as ImageNet are too computationally expensive, the authors could consider variants that are decently big and challenging, such as TinyImageNet. 2. For demonstration of experimental results,
The use of partial information decomposition in a learning framework with bivariate and trivariate implementations enables interpretable information processing at the per-neuron level, which is also new. The discussion based on a comparison between heuristic and optimization approaches demonstrates the potential of the interpretability of local learning framework in a task-relevant context and without the loss of performance compared to ANN trained with bp.
The title ‘What should a neuron aim for’ is broader than the scope explored in the current work, where a single-layer bivariate and trivariate local learning framework is studied. There is still a large gap between the interpretability of the model here and that of the neuron. The advantage of interpretability from an application perspective is not demonstrated or discussed, which is expected to be very limited by the single-layer simplicity here.
The idea of using information-theoretic loss functions to train individual neurons is interesting, with nice connection to biological neural networks and neuroscience. The novelty of adding a third input class representing lateral connections between neurons is intuitive and interesting and demonstrates better empirical performance than the bivariate model. The overall method and motivation are clearly presented. The experiments are well designed. Nice ablation study on the goal parameters.
Some parts of the methodology are not clear. Please see questions below. The PID-based goal functions and infomorphic neurons were originally proposed by Makkeh et al. 2023, and the contribution of this work is to introduce lateral connections as a third input class. However, in the Introduction, the authors claim the PID goal function to be one of the main contributions of this paper. The authors should explain more clearly the difference between this work and Makkeh et al. 2023, and define t
Code & Models
Videos
Taxonomy
TopicsNeural Networks and Applications
MethodsSparse Evolutionary Training