Gradient Scaling Effects in Adaptive Spectral PINNs for Stiff Nonlinear ODEs

TL;DR

This paper investigates how initial-condition gating functions influence gradient scaling and training stability in adaptive spectral PINNs applied to stiff nonlinear ODEs, revealing that IC embeddings significantly affect optimization outcomes.

Contribution

It characterizes the impact of IC gating functions on gradient scaling and training performance in adaptive spectral PINNs for stiff ODEs, highlighting their importance in model design.

Findings

Exponential gating yields lower error at moderate stiffness but shows variability across seeds.

Linear gating becomes preferable at higher stiffness levels.

Gradient scaling induced by IC embeddings significantly influences optimization conditioning.

Abstract

Physics-Informed Neural Networks (PINNs) often struggle to train reliably on stiff and oscillatory dynamical systems due to poor optimization conditioning. While prior work has emphasized representational remedies such as spectral parameterizations, the optimization implications of initial-condition (IC) embeddings in adaptive spectral PINNs have not been well characterized. In this work, we show that the choice of IC gating function induces explicit time-dependent gradient scaling, which interacts with spectral representations during training. Using a nonlinear stiff spring-pendulum ODE as a controlled benchmark, we compare exponential and linear IC gates in combination with fixed and adaptive Fourier spectral trunks. We observe stiffness-dependent changes in relative dominance for adaptive PINNs: at moderate stiffness ($k=20$), exponential gating often yields lower error but exhibits heterogeneous behavior across random seeds, whereas at higher stiffness ($k=60$), linear gating becomes preferable, with additional reversals observed at larger $k$. These trends hold for both relative $L^2$ error and maximum pointwise error and are confirmed by paired Wilcoxon signed-rank tests with Holm correction. Overall, our results demonstrate that IC embeddings are not a neutral design choice in PINNs: the induced gradient scaling materially shapes optimization conditioning in stiff regimes, with distinct sensitivity patterns in baseline and adaptive spectral models.