Convergence Analysis and Implicit Regularization of Feedback Alignment for Deep Linear Networks

TL;DR

This paper provides a theoretical analysis of Feedback Alignment, demonstrating convergence guarantees, exploring implicit regularization effects, and proposing initialization schemes to improve learning in deep linear networks.

Contribution

It offers the first convergence analysis for Feedback Alignment in deep linear networks and introduces initialization strategies that induce implicit regularization.

Findings

Feedback Alignment converges with specific rates in deep linear networks.

Certain initializations lead to implicit anti-regularization, hindering learning.

Proper initialization schemes can promote component-wise learning order.

Abstract

We theoretically analyze the Feedback Alignment (FA) algorithm, an efficient alternative to backpropagation for training neural networks. We provide convergence guarantees with rates for deep linear networks for both continuous and discrete dynamics. Additionally, we study incremental learning phenomena for shallow linear networks. Interestingly, certain specific initializations imply that negligible components are learned before the principal ones, thus potentially negatively affecting the effectiveness of such a learning algorithm; a phenomenon we classify as implicit anti-regularization. We also provide initialization schemes where the components of the problem are approximately learned by decreasing order of importance, thus providing a form of implicit regularization.