One-Time Soft Alignment Enables Resilient Learning without Weight Transport

TL;DR

This paper introduces a one-time soft alignment at initialization that allows deep networks to learn effectively without weight transport, improving stability, generalization, and robustness in a biologically plausible way.

Contribution

It proposes a simple initialization strategy enabling deep networks to perform well without ongoing weight symmetry, addressing limitations of feedback alignment.

Findings

Achieves comparable performance to backpropagation without weight transport.

Promotes smoother gradient flow and convergence to flatter minima.

Enhances adversarial robustness with moderate deviations from symmetry.

Abstract

Backpropagation is the cornerstone of deep learning, but its reliance on symmetric weight transport and global synchronization makes it computationally expensive and biologically implausible. Feedback alignment offers a promising alternative by approximating error gradients through fixed random feedback, thereby avoiding symmetric weight transport. However, this approach often struggles with poor learning performance and instability, especially in deep networks. Here, we show that a one-time soft alignment between forward and feedback weights at initialization enables deep networks to achieve performance comparable to backpropagation, without requiring weight transport during learning. This simple initialization condition guides stable error minimization in the loss landscape, improving network trainability. Spectral analyses further reveal that initial alignment promotes smoother gradient flow and convergence to flatter minima, resulting in better generalization and robustness. Notably, we also find that allowing moderate deviations from exact weight symmetry can improve adversarial robustness compared to standard backpropagation. These findings demonstrate that a simple initialization strategy can enable effective learning in deep networks in a biologically plausible and resource-efficient manner.