Block-local learning with probabilistic latent representations

TL;DR

This paper introduces a block-local learning method that enables parallel training of neural networks by dividing them into blocks with local losses, addressing the locking and weight transport problems of backpropagation.

Contribution

The paper proposes a novel block-local learning framework that allows parallel training and reduces dependency on weight transport, with a statistical interpretation of local errors.

Findings

Achieves state-of-the-art performance on various tasks.

Addresses locking and weight transport problems in neural training.

Enables scalable and distributed training of large models.

Abstract

The ubiquitous backpropagation algorithm requires sequential updates through the network introducing a locking problem. In addition, back-propagation relies on the transpose of forward weight matrices to compute updates, introducing a weight transport problem across the network. Locking and weight transport are problems because they prevent efficient parallelization and horizontal scaling of the training process. We propose a new method to address both these problems and scale up the training of large models. Our method works by dividing a deep neural network into blocks and introduces a feedback network that propagates the information from the targets backwards to provide auxiliary local losses. Forward and backward propagation can operate in parallel and with different sets of weights, addressing the problems of locking and weight transport. Our approach derives from a statistical interpretation of training that treats output activations of network blocks as parameters of probability distributions. The resulting learning framework uses these parameters to evaluate the agreement between forward and backward information. Error backpropagation is then performed locally within each block, leading to "block-local" learning. Several previously proposed alternatives to error backpropagation emerge as special cases of our model. We present results on a variety of tasks and architectures, demonstrating state-of-the-art performance using block-local learning. These results provide a new principled framework for training networks in a distributed setting.