Dynamic Spectral Backpropagation for Efficient Neural Network Training

TL;DR

Dynamic Spectral Backpropagation (DSBP) improves neural network training efficiency and robustness by projecting gradients onto principal eigenvectors, outperforming existing methods across various datasets and tasks.

Contribution

Introduces DSBP, a novel spectral gradient projection method with five extensions, supported by theoretical analysis, enhancing training efficiency and robustness.

Findings

DSBP outperforms SAM, LoRA, and MAML on multiple datasets.

Supports robustness, few-shot learning, and hardware efficiency.

Validated through extensive experiments and visualizations.

Abstract

Dynamic Spectral Backpropagation (DSBP) enhances neural network training under resource constraints by projecting gradients onto principal eigenvectors, reducing complexity and promoting flat minima. Five extensions are proposed, dynamic spectral inference, spectral architecture optimization, spectral meta learning, spectral transfer regularization, and Lie algebra inspired dynamics, to address challenges in robustness, fewshot learning, and hardware efficiency. Supported by a third order stochastic differential equation (SDE) and a PAC Bayes limit, DSBP outperforms Sharpness Aware Minimization (SAM), Low Rank Adaptation (LoRA), and Model Agnostic Meta Learning (MAML) on CIFAR 10, Fashion MNIST, MedMNIST, and Tiny ImageNet, as demonstrated through extensive experiments and visualizations. Future work focuses on scalability, bias mitigation, and ethical considerations.