Solving High-Dimensional PDEs Using Linearized Neural Networks

TL;DR

This paper investigates linearized neural networks for solving high-dimensional PDEs, revealing that collocation methods with least-squares solvers are more stable and accurate than variational approaches, and that random sampling of hidden parameters is unnecessary.

Contribution

The study provides a comprehensive numerical analysis of variational and collocation formulations for linearized neural networks in PDE solving, highlighting stability issues and proposing deterministic schemes for tanh activations.

Findings

Variational formulation leads to severely ill-conditioned linear systems.

Collocation methods with least-squares solvers are more stable and accurate.

Deterministic schemes for tanh activations achieve high accuracy without random sampling.

Abstract

Linearized shallow neural networks that are constructed by fixing the hidden-layer parameters have recently shown strong performance in solving partial differential equations (PDEs). Such models, widely used in the random feature method (RFM) and extreme learning machines (ELM), transform network training into a linear least-squares problem. In this paper, we conduct a numerical study of the variational (Galerkin) and collocation formulations for these linearized networks. Our numerical results reveal that, in the variational formulation, the associated linear systems are severely ill-conditioned, forming the primary computational bottleneck in scaling the neural network size, even when direct solvers are employed. In contrast, collocation methods combined with robust least-squares solvers exhibit better numerical stability and achieve higher accuracy as we increase neuron numbers. This behavior is consistently observed for both ReLU$^k$ and $\tanh$ activations, with $\tanh$ networks exhibiting even worse conditioning. Furthermore, we demonstrate that random sampling of the hidden layer parameters, commonly used in RFM and ELM, is not necessary for achieving high accuracy. For ReLU$^k$ activations, this follows from existing theory and is verified numerically in this paper, while for $\tanh$ activations, we introduce two deterministic schemes that achieve comparable accuracy.