Contrastive Forward-Forward: A Training Algorithm of Vision Transformer

TL;DR

This paper introduces Contrastive Forward-Forward, an improved training algorithm for Vision Transformers inspired by brain-like processes, achieving higher accuracy and faster convergence than the original Forward-Forward method and narrowing the gap with backpropagation.

Contribution

The paper extends the Forward-Forward algorithm to Vision Transformers and incorporates contrastive learning insights, significantly enhancing performance and convergence speed.

Findings

Up to 10% accuracy improvement over baseline Forward-Forward

Convergence accelerated by 5 to 20 times

Outperforms backpropagation under certain conditions

Abstract

Although backpropagation is widely accepted as a training algorithm for artificial neural networks, researchers are always looking for inspiration from the brain to find ways with potentially better performance. Forward-Forward is a novel training algorithm that is more similar to what occurs in the brain, although there is a significant performance gap compared to backpropagation. In the Forward-Forward algorithm, the loss functions are placed after each layer, and the updating of a layer is done using two local forward passes and one local backward pass. Forward-Forward is in its early stages and has been designed and evaluated on simple multi-layer perceptron networks to solve image classification tasks. In this work, we have extended the use of this algorithm to a more complex and modern network, namely the Vision Transformer. Inspired by insights from contrastive learning, we have attempted to revise this algorithm, leading to the introduction of Contrastive Forward-Forward. Experimental results show that our proposed algorithm performs significantly better than the baseline Forward-Forward leading to an increase of up to 10% in accuracy and accelerating the convergence speed by 5 to 20 times. Furthermore, if we take Cross Entropy as the baseline loss function in backpropagation, it will be demonstrated that the proposed modifications to the baseline Forward-Forward reduce its performance gap compared to backpropagation on Vision Transformer, and even outperforms it in certain conditions, such as inaccurate supervision.