Conducting Credit Assignment by Aligning Local Representations
Alexander G. Ororbia, Ankur Mali, Daniel Kifer, and C. Lee Giles

TL;DR
The paper introduces Local Representation Alignment (LRA), a robust training method for deep networks that overcomes common issues of back-propagation, works with various nonlinearities, and performs well on challenging datasets.
Contribution
LRA is a novel training procedure that is less sensitive to initializations, adaptable to different architectures and nonlinearities, and can train networks from null initialization.
Findings
LRA successfully trains networks on MNIST and Fashion MNIST.
LRA outperforms target propagation and feedback alignment.
LRA is effective even with null weight initialization.
Abstract
Using back-propagation and its variants to train deep networks is often problematic for new users. Issues such as exploding gradients, vanishing gradients, and high sensitivity to weight initialization strategies often make networks difficult to train, especially when users are experimenting with new architectures. Here, we present Local Representation Alignment (LRA), a training procedure that is much less sensitive to bad initializations, does not require modifications to the network architecture, and can be adapted to networks with highly nonlinear and discrete-valued activation functions. Furthermore, we show that one variation of LRA can start with a null initialization of network weights and still successfully train networks with a wide variety of nonlinearities, including tanh, ReLU-6, softplus, signum and others that may draw their inspiration from biology. A comprehensive set…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsDomain Adaptation and Few-Shot Learning · Ferroelectric and Negative Capacitance Devices · Machine Learning and ELM
