Scalable Forward-Forward Algorithm

TL;DR

This paper introduces a scalable Forward-Forward algorithm that trains neural networks layer-by-layer without backpropagation, achieving comparable performance and improved modularity on modern architectures.

Contribution

It extends the Forward-Forward algorithm to convolutional networks like MobileNetV3 and ResNet18, enabling layer-wise training without backpropagation.

Findings

Achieves performance comparable to backpropagation on standard benchmarks.

Hybrid training with block-wise backpropagation outperforms full backpropagation.

Demonstrates effectiveness on small datasets and transfer learning scenarios.

Abstract

We propose a scalable Forward-Forward (FF) algorithm that eliminates the need for backpropagation by training each layer separately. Unlike backpropagation, FF avoids backward gradients and can be more modular and memory efficient, making it appealing for large networks. We extend FF to modern convolutional architectures, such as MobileNetV3 and ResNet18, by introducing a new way to compute losses for convolutional layers. Experiments show that our method achieves performance comparable to standard backpropagation. Furthermore, when we divide the network into blocks, such as the residual blocks in ResNet, and apply backpropagation only within each block, but not across blocks, our hybrid design tends to outperform backpropagation baselines while maintaining a similar training speed. Finally, we present experiments on small datasets and transfer learning that confirm the adaptability of our method.