Principled Training of Neural Networks with Direct Feedback Alignment

TL;DR

This paper introduces best practices for training neural networks using direct feedback alignment, addressing scalability issues and proposing solutions to improve its performance on complex tasks.

Contribution

It provides a set of empirically justified best practices for direct feedback alignment and analyzes a bottleneck affecting alignment in narrow layers.

Findings

Alignment angles are crucial for effective training.

Narrow layers create a bottleneck hindering feedback alignment.

Proper practices can improve scalability of feedback alignment methods.

Abstract

The backpropagation algorithm has long been the canonical training method for neural networks. Modern paradigms are implicitly optimized for it, and numerous guidelines exist to ensure its proper use. Recently, synthetic gradients methods -where the error gradient is only roughly approximated - have garnered interest. These methods not only better portray how biological brains are learning, but also open new computational possibilities, such as updating layers asynchronously. Even so, they have failed to scale past simple tasks like MNIST or CIFAR-10. This is in part due to a lack of standards, leading to ill-suited models and practices forbidding such methods from performing to the best of their abilities. In this work, we focus on direct feedback alignment and present a set of best practices justified by observations of the alignment angles. We characterize a bottleneck effect that prevents alignment in narrow layers, and hypothesize it may explain why feedback alignment methods have yet to scale to large convolutional networks.