Self-Contrastive Forward-Forward Algorithm

TL;DR

The paper introduces the Self-Contrastive Forward-Forward (SCFF) algorithm, enhancing the Forward-Forward method with contrastive learning to achieve competitive performance on benchmarks while maintaining suitability for resource-constrained, decentralized systems.

Contribution

It proposes the SCFF algorithm that improves Forward-Forward training with self-contrastive methods, enabling better unsupervised learning and extending applicability to recurrent neural networks.

Findings

SCFF outperforms existing unsupervised local learning algorithms on benchmarks.

The method is effective across multiple datasets including MNIST, CIFAR-10, STL-10, and Tiny ImageNet.

SCFF enables high-accuracy, real-time learning on resource-constrained devices.

Abstract

Agents that operate autonomously benefit from lifelong learning capabilities. However, compatible training algorithms must comply with the decentralized nature of these systems, which imposes constraints on both the parameter counts and the computational resources. The Forward-Forward (FF) algorithm is one of these. FF relies only on feedforward operations, the same used for inference, for optimizing layer-wise objectives. This purely forward approach eliminates the need for transpose operations required in traditional backpropagation. Despite its potential, FF has failed to reach state-of-the-art performance on most standard benchmark tasks, in part due to unreliable negative data generation methods for unsupervised learning. In this work, we propose the Self-Contrastive Forward-Forward (SCFF) algorithm, a competitive training method aimed at closing this performance gap. Inspired by standard self-supervised contrastive learning for vision tasks, SCFF generates positive and negative inputs applicable across various datasets. The method demonstrates superior performance compared to existing unsupervised local learning algorithms on several benchmark datasets, including MNIST, CIFAR-10, STL-10, and Tiny ImageNet. We extend FF's application to training recurrent neural networks, expanding its utility to sequential data tasks. These findings pave the way for high-accuracy, real-time learning on resource-constrained edge devices.