Learning by the F-adjoint

TL;DR

This paper explores the F-adjoint method as a new theoretical framework for neural network training, leading to a local learning rule that improves performance over standard back-propagation, demonstrated on MNIST datasets.

Contribution

It introduces an equilibrium F-adjoint process derived from neural dynamical models combined with gradient descent, offering a novel local learning rule for deep networks.

Findings

Significant performance improvements on MNIST and Fashion-MNIST datasets.

Development of a local learning rule based on the F-adjoint framework.

Experimental validation showing advantages over standard back-propagation.

Abstract

A recent paper by Boughammoura (2023) describes the back-propagation algorithm in terms of an alternative formulation called the F-adjoint method. In particular, by the F-adjoint algorithm the computation of the loss gradient, with respect to each weight within the network, is straightforward and can simply be done. In this work, we develop and investigate this theoretical framework to improve some supervised learning algorithm for feed-forward neural network. Our main result is that by introducing some neural dynamical model combined by the gradient descent algorithm, we derived an equilibrium F-adjoint process which yields to some local learning rule for deep feed-forward networks setting. Experimental results on MNIST and Fashion-MNIST datasets, demonstrate that the proposed approach provide a significant improvements on the standard back-propagation training procedure.