Gradient target propagation

TL;DR

This paper introduces a novel learning rule called gradient target propagation that estimates neuron targets to optimize neural network training, encompassing backpropagation and Hebbian learning, with competitive results on standard datasets.

Contribution

The paper presents a new learning rule for neural networks that generalizes backpropagation and Hebbian learning, along with a weight initialization technique, demonstrating comparable performance to backpropagation.

Findings

Achieved similar accuracy to backpropagation on MNIST, Fashion-MNIST, and CIFAR-10.

Provides a theoretical framework unifying backpropagation and Hebbian learning.

Offers a new weight initialization method for neural networks.

Abstract

We report a learning rule for neural networks that computes how much each neuron should contribute to minimize a giving cost function via the estimation of its target value. By theoretical analysis, we show that this learning rule contains backpropagation, Hebian learning, and additional terms. We also give a general technique for weights initialization. Our results are at least as good as those obtained with backpropagation. The neural networks are trained and tested in three problems: MNIST, MNIST-Fashion, and CIFAR-10 datasets. The associated code is available at https://github.com/tiago939/target.