Towards Interpretable Deep Local Learning with Successive Gradient Reconciliation

TL;DR

This paper introduces a novel local training strategy for neural networks that successively reconciles gradients between modules, improving performance and reducing memory use without relying on global back-propagation.

Contribution

It provides the first theoretical analysis of gradient reconciliation in local learning and proposes a new method that enhances local training with significant empirical gains.

Findings

Achieves competitive performance with global BP on ImageNet.

Reduces memory consumption by over 40%.

Improves training stability and convergence in local learning.

Abstract

Relieving the reliance of neural network training on a global back-propagation (BP) has emerged as a notable research topic due to the biological implausibility and huge memory consumption caused by BP. Among the existing solutions, local learning optimizes gradient-isolated modules of a neural network with local errors and has been proved to be effective even on large-scale datasets. However, the reconciliation among local errors has never been investigated. In this paper, we first theoretically study non-greedy layer-wise training and show that the convergence cannot be assured when the local gradient in a module w.r.t. its input is not reconciled with the local gradient in the previous module w.r.t. its output. Inspired by the theoretical result, we further propose a local training strategy that successively regularizes the gradient reconciliation between neighboring modules without breaking gradient isolation or introducing any learnable parameters. Our method can be integrated into both local-BP and BP-free settings. In experiments, we achieve significant performance improvements compared to previous methods. Particularly, our method for CNN and Transformer architectures on ImageNet is able to attain a competitive performance with global BP, saving more than 40% memory consumption.