SAL: Selective Adaptive Learning for Backpropagation-Free Training with Sparsification

TL;DR

SAL introduces a biologically inspired, backpropagation-free training method that employs selective parameter activation and adaptive partitioning, reducing gradient interference and enabling scalable deep learning.

Contribution

It proposes a novel training approach combining selective activation and adaptive partitioning, addressing backpropagation limitations and enabling training of very deep and large-scale models.

Findings

Competitive convergence rates across 10 benchmarks

Effective in training deep networks up to 128 layers

Maintains accuracy in models with up to 1 billion parameters

Abstract

Standard deep learning relies on Backpropagation (BP), which is constrained by biologically implausible weight symmetry and suffers from significant gradient interference within dense representations. To mitigate these bottlenecks, we propose Selective Adaptive Learning (SAL), a training method that combines selective parameter activation with adaptive area partitioning. Specifically, SAL decomposes the parameter space into mutually exclusive, sample-dependent regions. This decoupling mitigates gradient interference across divergent semantic patterns and addresses explicit weight symmetry requirements through our refined feedback alignment. Empirically, SAL demonstrates competitive convergence rates, leading to improved classification performance across 10 standard benchmarks. Additionally, SAL achieves numerical consistency and competitive accuracy even in deep regimes (up to 128 layers) and large-scale models (up to 1B parameters). Our approach is loosely inspired by biological learning mechanisms, offering a plausible alternative that contributes to the study of scalable neural network training.