A Contrastive Symmetric Forward-Forward Algorithm (SFFA) for Continual Learning Tasks

TL;DR

This paper introduces the Symmetric Forward-Forward Algorithm (SFFA), an improved version of FFA that enhances training stability and generalization by symmetrizing the loss landscape, and explores its benefits for continual learning tasks.

Contribution

The work proposes SFFA, a novel symmetric modification of FFA that improves convergence and generalization, and demonstrates its effectiveness in continual learning scenarios.

Findings

SFFA achieves higher accuracy than FFA on image classification benchmarks.

Layer-wise training with SFFA facilitates continual learning by reducing catastrophic forgetting.

SFFA provides a more symmetric and stable training process compared to the original FFA.

Abstract

The so-called Forward-Forward Algorithm (FFA) has recently gained momentum as an alternative to the conventional back-propagation algorithm for neural network learning, yielding competitive performance across various modeling tasks. By replacing the backward pass of gradient back-propagation with two contrastive forward passes, the FFA avoids several shortcomings undergone by its predecessor (e.g., vanishing/exploding gradient) by enabling layer-wise training heuristics. In classification tasks, this contrastive method has been proven to effectively create a latent sparse representation of the input data, ultimately favoring discriminability. However, FFA exhibits an inherent asymmetric gradient behavior due to an imbalanced loss function between positive and negative data, adversely impacting on the model's generalization capabilities and leading to an accuracy degradation. To address this issue, this work proposes the Symmetric Forward-Forward Algorithm (SFFA), a novel modification of the original FFA which partitions each layer into positive and negative neurons. This allows the local fitness function to be defined as the ratio between the activation of positive neurons and the overall layer activity, resulting in a symmetric loss landscape during the training phase. To evaluate the enhanced convergence of our method, we conduct several experiments using multiple image classification benchmarks, comparing the accuracy of models trained with SFFA to those trained with its FFA counterpart. As a byproduct of this reformulation, we explore the advantages of using a layer-wise training algorithm for Continual Learning (CL) tasks. The specialization of neurons and the sparsity of their activations induced by layer-wise training algorithms enable efficient CL strategies that incorporate new knowledge (classes) into the neural network, while preventing catastrophic forgetting of previously...