Multigrade Neural Network Approximation

TL;DR

This paper introduces multigrade deep learning (MGDL), a hierarchical training framework for deep neural networks that guarantees approximation error reduction through a grade-by-grade residual learning approach.

Contribution

It provides the first rigorous theoretical proof that grade-wise training of deep networks achieves vanishing approximation error, supported by operator-theoretic foundations and numerical experiments.

Findings

MGDL guarantees residuals decrease across grades for any continuous target function.

Theoretical proof shows convergence of residuals to zero in fixed-width ReLU schemes.

Numerical experiments validate the theoretical approximation guarantees.

Abstract

We study multigrade deep learning (MGDL) as a principled framework for structured error refinement in deep neural networks. While the approximation power of neural networks is now relatively well understood, training very deep architectures remains challenging due to highly non-convex and often ill-conditioned optimization landscapes. In contrast, for relatively shallow networks, most notably one-hidden-layer $\texttt{ReLU}$ models, training admits convex reformulations with global guarantees, motivating learning paradigms that improve stability while scaling to depth. MGDL builds upon this insight by training deep networks grade by grade: previously learned grades are frozen, and each new residual block is trained solely to reduce the remaining approximation error, yielding an interpretable and stable hierarchical refinement process. We develop an operator-theoretic foundation for MGDL and prove that, for any continuous target function, there exists a fixed-width multigrade $\texttt{ReLU}$ scheme whose residuals decrease strictly across grades and converge uniformly to zero. To the best of our knowledge, this work provides the first rigorous theoretical guarantee that grade-wise training yields provable vanishing approximation error in deep networks. Numerical experiments further illustrate the theoretical results.