Forward Only Learning for Orthogonal Neural Networks of any Depth

TL;DR

This paper introduces FOTON, a forward-only training algorithm for orthogonal neural networks that scales to any depth and outperforms previous methods like PEPITA, reducing computational costs without requiring backpropagation.

Contribution

The paper presents FOTON, a novel forward-only training method for orthogonal neural networks that overcomes scalability issues of prior approaches and bridges the gap with backpropagation.

Findings

FOTON outperforms PEPITA in training depth and accuracy.

FOTON enables training of neural networks of any depth without backward passes.

Performance on convolutional networks suggests broader applicability.

Abstract

Backpropagation is still the de facto algorithm used today to train neural networks. With the exponential growth of recent architectures, the computational cost of this algorithm also becomes a burden. The recent PEPITA and forward-only frameworks have proposed promising alternatives, but they failed to scale up to a handful of hidden layers, yet limiting their use. In this paper, we first analyze theoretically the main limitations of these approaches. It allows us the design of a forward-only algorithm, which is equivalent to backpropagation under the linear and orthogonal assumptions. By relaxing the linear assumption, we then introduce FOTON (Forward-Only Training of Orthogonal Networks) that bridges the gap with the backpropagation algorithm. Experimental results show that it outperforms PEPITA, enabling us to train neural networks of any depth, without the need for a backward pass. Moreover its performance on convolutional networks clearly opens up avenues for its application to more advanced architectures. The code is open-sourced at https://github.com/p0lcAi/FOTON .