Convergence Analysis of Block Newton Methods for 1D Shallow Neural Network Approximation

TL;DR

This paper provides a theoretical analysis of the local convergence of block Newton methods for one-dimensional shallow neural network approximation, including a novel reduced BN method that adaptively reduces parameters during optimization.

Contribution

It introduces and analyzes the convergence of the reduced BN method, which adaptively reduces parameters, for neural network approximation and diffusion-reaction problems.

Findings

Proves local convergence of BN and rBN methods under certain assumptions.

Demonstrates the effectiveness of rBN in reducing parameters during optimization.

Applies the methods to diffusion-reaction problems and least-squares approximation.

Abstract

This paper analyzes local convergence of the block Newton (BN) method introduced in [5, 6] for one-dimensional shallow neural network approximation to functions and diffusion-reaction problems. The BN method consists of the 2x2 block nonlinear Gauss-Seidel, linear Gauss-Seidel, or Jacobi method for outer iteration and the Newton method for inner iteration. The blocks are corresponding to the linear and the nonlinear parameters. Under some reasonable assumptions, we establish local convergence of the BN methods as well as the reduced BN (rBN) method for one-dimensional diffusion-reaction problems and least-squares function approximation. Unlike common optimization methods, the rBN allows for the reduction of the number of parameters during the optimization process when some neurons contribute little to the approximation or are at nearly optimal locations.