Successive Affine Learning for Deep Neural Networks

TL;DR

This paper proposes a successive affine learning (SAL) model for deep neural networks that simplifies training by solving convex problems for each layer, leading to improved performance over traditional methods.

Contribution

The SAL model introduces a layer-wise convex optimization approach for training DNNs, inspired by human education, and provides theoretical convergence guarantees.

Findings

SAL outperforms traditional deep learning models in numerical tests.

The model establishes Pythagorean and Parseval identities for the generated system.

Convergence theorem shows either finite termination or error norms decrease to a limit.

Abstract

This paper introduces a successive affine learning (SAL) model for constructing deep neural networks (DNNs). Traditionally, a DNN is built by solving a non-convex optimization problem. It is often challenging to solve such a problem numerically due to its non-convexity and having a large number of layers. To address this challenge, inspired by the human education system, the multi-grade deep learning (MGDL) model was recently initiated by the author of this paper. The MGDL model learns a DNN in several grades, in each of which one constructs a shallow DNN consisting of a relatively small number of layers. The MGDL model still requires solving several non-convex optimization problems. The proposed SAL model mutates from the MGDL model. Noting that each layer of a DNN consists of an affine map followed by an activation function, we propose to learn the affine map by solving a quadratic/convex optimization problem which involves the activation function only {\it after} the weight matrix and the bias vector for the current layer have been trained. In the context of function approximation, for a given function the SAL model generates an expansion of the function with adaptive basis functions in the form of DNNs. We establish the Pythagorean identity and the Parseval identity for the system generated by the SAL model. Moreover, we provide a convergence theorem of the SAL process in the sense that either it terminates after a finite number of grades or the norms of its optimal error functions strictly decrease to a limit as the grade number increases to infinity. Furthermore, we present numerical examples of proof of concept which demonstrate that the proposed SAL model significantly outperforms the traditional deep learning model.