Exploring the Dimensions of a Variational Neuron

TL;DR

This paper introduces EVE, a variational neuron model that incorporates local probabilistic structure at the neuron level, enabling detailed study of how internal latent dimensions influence neuron behavior and network performance.

Contribution

It presents a novel variational neuron framework with local probabilistic structure, allowing exploration of internal latent dimensions and their impact on neural operation.

Findings

Latent dimensionality affects neuron operating regimes.

Neuron-level variables are measurable and related to downstream behavior.

Structural properties influence internal neuron dynamics.

Abstract

We introduce EVE (Elemental Variational Expanse), a variational distributional neuron formulated as a local probabilistic computational unit with an explicit prior, an amortized posterior, and unit-level variational regularization. In most modern architectures, uncertainty is modeled through global latent variables or parameter uncertainty, while the computational unit itself remains scalar. EVE instead relocates probabilistic structure to the neuron level, making it locally observable and controllable. In this paper, the term dimensions refers primarily to the neuron's internal latent dimensionality, denoted by k. We study how varying k, from the atomic case k = 1 to higher-dimensional latent spaces, changes the neuron's learned operating regime. We then examine how this main axis interacts with two additional structural properties: local capacity control and temporal persistence through a neuron-level autoregressive extension. To support this study, EVE is instrumented with internal diagnostics and constraints, including effective KL, a target band on mu^2, out-of-band fractions, and indicators of drift and collapse. Across selected forecasting and tabular settings, we show that latent dimensionality, control, and temporal extension shape the neuron's internal regime, and that some neuron-level variables are measurable, informative, and related to downstream behavior. Overall, the paper provides an experimentally grounded first map of the design space opened by a variational neuron.