Biology-inspired joint distribution neurons based on Hierarchical Correlation Reconstruction allowing for multidirectional propagation of values and densities

TL;DR

This paper introduces biologically inspired joint distribution neurons based on Hierarchical Correlation Reconstruction, enabling multidirectional information flow, probabilistic modeling, and local training methods to improve artificial neural networks.

Contribution

It extends the KAN approach to include joint distribution modeling with multidirectional propagation, probabilistic outputs, and local training methods, aligning artificial neurons more closely with biological properties.

Findings

Proposed joint distribution neurons can propagate distributions in multiple directions.

The approach allows for local training methods beyond backpropagation.

Neurons can predict and propagate distributions of moments like mean and variance.

Abstract

Recently a million of biological neurons (BNN) has turned out better from modern RL methods in playing Pong~\cite{RL}, reminding they are still qualitatively superior e.g. in learning, flexibility and robustness - suggesting to try to improve current artificial e.g. MLP/KAN for better agreement with biological. There is proposed extension of KAN approach to neurons containing model of local joint distribution: $\rho(\mathbf{x})=\sum_{\mathbf{j}\in B} a_\mathbf{j} f_\mathbf{j}(\mathbf{x})$ for $\mathbf{x} \in [0,1]^d$, adding interpretation and information flow control to KAN, and allowing to gradually add missing 3 basic properties of biological: 1) biological axons propagate in both directions~\cite{axon}, while current artificial are focused on unidirectional propagation - joint distribution neurons can repair by substituting some variables to get conditional values/distributions for the remaining. 2) Animals show risk avoidance~\cite{risk} requiring to process variance, and generally real world rather needs probabilistic models - the proposed can predict and propagate also distributions as vectors of moments: (expected value, variance) or higher. 3) biological neurons require local training, and beside backpropagation, the proposed allows many additional ways, like direct training, through tensor decomposition, or finally local and promising: information bottleneck. Proposed approach is very general, can be also used as extension of softmax in embeddings of e.g. transformer or JEPA, suggesting interpretation that features are mixed moments of joint density of real-world properties.