Direct Feedback Alignment Provides Learning in Deep Neural Networks

TL;DR

This paper introduces direct feedback alignment, a biologically plausible learning method for deep neural networks that propagates errors through fixed random feedback connections, achieving comparable performance to back-propagation.

Contribution

It demonstrates that error signals can be propagated directly from output to hidden layers using fixed random feedback, eliminating the need for symmetric weights and back-propagation.

Findings

Achieves zero training error in deep and convolutional networks without back-propagation.

Test performance on MNIST and CIFAR is comparable to traditional back-propagation.

With dropout, reaches 1.45% error on MNIST.

Abstract

Artificial neural networks are most commonly trained with the back-propagation algorithm, where the gradient for learning is provided by back-propagating the error, layer by layer, from the output layer to the hidden layers. A recently discovered method called feedback-alignment shows that the weights used for propagating the error backward don't have to be symmetric with the weights used for propagation the activation forward. In fact, random feedback weights work evenly well, because the network learns how to make the feedback useful. In this work, the feedback alignment principle is used for training hidden layers more independently from the rest of the network, and from a zero initial condition. The error is propagated through fixed random feedback connections directly from the output layer to each hidden layer. This simple method is able to achieve zero training error even in convolutional networks and very deep networks, completely without error back-propagation. The method is a step towards biologically plausible machine learning because the error signal is almost local, and no symmetric or reciprocal weights are required. Experiments show that the test performance on MNIST and CIFAR is almost as good as those obtained with back-propagation for fully connected networks. If combined with dropout, the method achieves 1.45% error on the permutation invariant MNIST task.