A comparative study of back propagation and its alternatives on multilayer perceptrons

TL;DR

This paper compares backpropagation with alternative training algorithms for multilayer perceptrons, analyzing their stability, similarity of predictions, and neuron behavior, and introduces a new variation of one alternative algorithm.

Contribution

It provides a comparative analysis of backpropagation and its alternatives, including a new variation of one of these algorithms, focusing on stability and neuron similarity in MLPs.

Findings

Backpropagation remains effective for MLP training.

Alternatives show comparable stability and prediction similarity.

A new variation of an alternative algorithm improves certain aspects.

Abstract

The de facto algorithm for training the back pass of a feedforward neural network is backpropagation (BP). The use of almost-everywhere differentiable activation functions made it efficient and effective to propagate the gradient backwards through layers of deep neural networks. However, in recent years, there has been much research in alternatives to backpropagation. This analysis has largely focused on reaching state-of-the-art accuracy in multilayer perceptrons (MLPs) and convolutional neural networks (CNNs). In this paper, we analyze the stability and similarity of predictions and neurons in MLPs and propose a new variation of one of the algorithms.