Emergent representations in networks trained with the Forward-Forward algorithm

TL;DR

This paper demonstrates that networks trained with the biologically plausible Forward-Forward algorithm develop sparse, category-specific internal representations similar to cortical ensembles, unlike traditional Backpropagation.

Contribution

It shows that the Forward-Forward algorithm can produce biologically plausible, sparse neural representations, bridging the gap between artificial and biological neural networks.

Findings

Forward-Forward-trained networks develop sparse, category-specific ensembles.

Such sparse representations are absent in models trained with Backpropagation.

The same objective used in Forward-Forward can induce similar patterns in Backpropagation-trained models.

Abstract

The Backpropagation algorithm has often been criticised for its lack of biological realism. In an attempt to find a more biologically plausible alternative, the recently introduced Forward-Forward algorithm replaces the forward and backward passes of Backpropagation with two forward passes. In this work, we show that the internal representations obtained by the Forward-Forward algorithm can organise into category-specific ensembles exhibiting high sparsity -- composed of a low number of active units. This situation is reminiscent of what has been observed in cortical sensory areas, where neuronal ensembles are suggested to serve as the functional building blocks for perception and action. Interestingly, while this sparse pattern does not typically arise in models trained with standard Backpropagation, it can emerge in networks trained with Backpropagation on the same objective proposed for the Forward-Forward algorithm.