A neuron-wise subspace correction method for the finite neuron method

TL;DR

This paper introduces a neuron-wise subspace correction algorithm for neural network-based PDE solvers, improving training efficiency and accuracy by optimizing linear and nonlinear components separately.

Contribution

It develops a novel subspace correction method with an optimal preconditioner, enhancing training stability and convergence for neural PDE approximations.

Findings

Better approximation accuracy than existing methods

Uniform training iterations across neurons due to preconditioning

Superlinear convergence in single neuron optimization

Abstract

In this paper, we propose a novel algorithm called Neuron-wise Parallel Subspace Correction Method (NPSC) for the finite neuron method that approximates numerical solutions of partial differential equations (PDEs) using neural network functions. Despite extremely extensive research activities in applying neural networks for numerical PDEs, there is still a serious lack of effective training algorithms that can achieve adequate accuracy, even for one-dimensional problems. Based on recent results on the spectral properties of linear layers and landscape analysis for single neuron problems, we develop a special type of subspace correction method that optimizes the linear layer and each neuron in the nonlinear layer separately. An optimal preconditioner that resolves the ill-conditioning of the linear layer is presented for one-dimensional problems, so that the linear layer is trained in a uniform number of iterations with respect to the number of neurons. In each single neuron problem, a good local minimum that avoids flat energy regions is found by a superlinearly convergent algorithm. Numerical experiments on function approximation problems and PDEs demonstrate better performance of the proposed method than other gradient-based methods.