Fast, Convex and Conditioned Network for Multi-Fidelity Vectors and Stiff Univariate Differential Equations

TL;DR

This paper introduces Shifted Gaussian Encoding to improve the conditioning of neural PDE solvers, significantly enhancing accuracy and convergence in multi-fidelity and stiff problems by addressing ill-conditioning issues.

Contribution

It proposes a simple activation filtering step that increases matrix rank and expressivity, extending the solvable range of challenging PDEs and outperforming deep networks in accuracy and speed.

Findings

Extended the solvable Peclet number range by over two orders of magnitude.

Achieved up to six orders lower error in multi-frequency function learning.

Fitted high-fidelity image vectors more accurately and faster than large deep networks.

Abstract

Accuracy in neural PDE solvers often breaks down not because of limited expressivity, but due to poor optimisation caused by ill-conditioning, especially in multi-fidelity and stiff problems. We study this issue in Physics-Informed Extreme Learning Machines (PIELMs), a convex variant of neural PDE solvers, and show that asymptotic components in governing equations can produce highly ill-conditioned activation matrices, severely limiting convergence. We introduce Shifted Gaussian Encoding, a simple yet effective activation filtering step that increases matrix rank and expressivity while preserving convexity. Our method extends the solvable range of Peclet numbers in steady advection-diffusion equations by over two orders of magnitude, achieves up to six orders lower error on multi-frequency function learning, and fits high-fidelity image vectors more accurately and faster than deep networks with over a million parameters. This work highlights that conditioning, not depth, is often the bottleneck in scientific neural solvers and that simple architectural changes can unlock substantial gains.