Self-adaptive deep neural network: Numerical approximation to functions and PDEs

TL;DR

This paper introduces a self-adaptive algorithm for neural network design that iteratively enhances the network structure based on estimators, enabling efficient approximation of complex functions and PDE solutions.

Contribution

The paper presents a novel adaptive network enhancement (ANE) method that automatically determines network size and structure for optimal approximation of complex functions and PDEs.

Findings

Automatically designs nearly minimal neural networks for complex functions.

Effectively approximates solutions with sharp transitions and discontinuities.

Provides a systematic approach for network initialization and enhancement.

Abstract

Designing an optimal deep neural network for a given task is important and challenging in many machine learning applications. To address this issue, we introduce a self-adaptive algorithm: the adaptive network enhancement (ANE) method, written as loops of the form train, estimate and enhance. Starting with a small two-layer neural network (NN), the step train is to solve the optimization problem at the current NN; the step estimate is to compute a posteriori estimator/indicators using the solution at the current NN; the step enhance is to add new neurons to the current NN. Novel network enhancement strategies based on the computed estimator/indicators are developed in this paper to determine how many new neurons and when a new layer should be added to the current NN. The ANE method provides a natural process for obtaining a good initialization in training the current NN; in addition, we introduce an advanced procedure on how to initialize newly added neurons for a better approximation. We demonstrate that the ANE method can automatically design a nearly minimal NN for learning functions exhibiting sharp transitional layers as well as discontinuous solutions of hyperbolic partial differential equations.