Chebyshev Moment Regularization (CMR): Condition-Number Control with Moment Shaping

TL;DR

Chebyshev Moment Regularization (CMR) is a novel loss function that improves neural network training stability and accuracy by controlling layer spectra through spectral shaping and condition number optimization.

Contribution

The paper introduces CMR, a new architecture-agnostic regularization method that directly optimizes layer spectra and condition numbers, with proven theoretical properties and practical effectiveness.

Findings

Reduces mean layer condition numbers by ~1000 times in 5 epochs

Restores test accuracy from ~10% to ~86% on MNIST with adversarial stress

Increases average gradient magnitude during training

Abstract

We introduce \textbf{Chebyshev Moment Regularization (CMR)}, a simple, architecture-agnostic loss that directly optimizes layer spectra. CMR jointly controls spectral edges via a log-condition proxy and shapes the interior via Chebyshev moments, with a decoupled, capped mixing rule that preserves task gradients. We prove strictly monotone descent for the condition proxy, bounded moment gradients, and orthogonal invariance. In an adversarial ``$\kappa$-stress'' setting (MNIST, 15-layer MLP), \emph{compared to vanilla training}, CMR reduces mean layer condition numbers by $\sim\!10^3$ (from $\approx3.9\!\times\!10^3$ to $\approx3.4$ in 5 epochs), increases average gradient magnitude, and restores test accuracy ( $\approx10\%\!\to\!\approx86\%$ ). These results support \textbf{optimization-driven spectral preconditioning}: directly steering models toward well-conditioned regimes for stable, accurate learning.