A theory for the sparsity emerged in the Forward Forward algorithm

TL;DR

This paper develops a theoretical framework explaining the high sparsity observed in the forward-forward algorithm, supported by experiments on MNIST that validate the predicted sparsity changes.

Contribution

It introduces two theorems predicting sparsity changes in the forward-forward algorithm, providing a theoretical understanding of the phenomenon.

Findings

The theorems accurately predict sparsity changes in different scenarios.

Experimental results on MNIST support the theoretical predictions.

The theory explains the emergence of sparsity in the forward-forward algorithm.

Abstract

This report explores the theory that explains the high sparsity phenomenon \citep{tosato2023emergent} observed in the forward-forward algorithm \citep{hinton2022forward}. The two theorems proposed predict the sparsity changes of a single data point's activation in two cases: Theorem \ref{theorem:1}: Decrease the goodness of the whole batch. Theorem \ref{theorem:2}: Apply the complete forward forward algorithm to decrease the goodness for negative data and increase the goodness for positive data. The theory aligns well with the experiments tested on the MNIST dataset.