Stochastic Forward-Forward Learning through Representational Dimensionality Compression

TL;DR

This paper introduces a novel stochastic Forward-Forward learning algorithm that uses representational dimensionality compression to improve neural network training, emphasizing biological plausibility and potential for neuromorphic computing.

Contribution

It proposes a new goodness function based on effective dimensionality, enabling contrastive learning without negative samples and highlighting the constructive role of noise.

Findings

Achieves competitive performance with non-backpropagation methods

Noise enhances generalization and inference quality

Structured representations are promoted through dimensionality compression

Abstract

The Forward-Forward (FF) learning algorithm provides a bottom-up alternative to backpropagation (BP) for training neural networks, relying on a layer-wise "goodness" function with well-designed negative samples for contrastive learning. Existing goodness functions are typically defined as the sum of squared postsynaptic activations, neglecting correlated variability between neurons. In this work, we propose a novel goodness function termed dimensionality compression that uses the effective dimensionality (ED) of fluctuating neural responses to incorporate second-order statistical structure. Our objective minimizes ED for noisy copies of individual inputs while maximizing it across the sample distribution, promoting structured representations without the need to prepare negative samples.We demonstrate that this formulation achieves competitive performance compared to other non-BP methods. Moreover, we show that noise plays a constructive role that can enhance generalization and improve inference when predictions are derived from the mean of squared output, which is equivalent to making predictions based on an energy term. Our findings contribute to the development of more biologically plausible learning algorithms and suggest a natural fit for neuromorphic computing, where stochasticity is a computational resource rather than a nuisance. The code is available at https://github.com/ZhichaoZhu/StochasticForwardForward