Variational Distributional Neuron
Yves Ruffenach

TL;DR
This paper introduces a novel neural compute unit called the variational distributional neuron, which models activations as distributions using a VAE framework, enabling explicit uncertainty representation within individual neurons.
Contribution
It proposes a new neuron model that incorporates a prior, posterior, and ELBO, transforming neurons into distributional units with controllable uncertainty and temporal dynamics.
Findings
Neurons can be modeled as distributional units with internal constraints.
The approach allows monitoring and tuning of information carried by neurons.
Extension to autoregressive priors enhances temporal modeling.
Abstract
We propose a proof of concept for a variational distributional neuron: a compute unit formulated as a VAE brick, explicitly carrying a prior, an amortized posterior and a local ELBO. The unit is no longer a deterministic scalar but a distribution: computing is no longer about propagating values, but about contracting a continuous space of possibilities under constraints. Each neuron parameterizes a posterior, propagates a reparameterized sample and is regularized by the KL term of a local ELBO - hence, the activation is distributional. This "contraction" becomes testable through local constraints and can be monitored via internal measures. The amount of contextual information carried by the unit, as well as the temporal persistence of this information, are locally tuned by distinct constraints. This proposal addresses a structural tension: in sequential generation, causality is…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsGenerative Adversarial Networks and Image Synthesis · Embodied and Extended Cognition · Neural dynamics and brain function
