Weight-Parameterization in Continuous Time Deep Neural Networks for Surrogate Modeling

TL;DR

This paper explores polynomial basis-based weight parameterizations in continuous-time neural networks, demonstrating improved training stability and efficiency while maintaining high accuracy in surrogate modeling of physical systems.

Contribution

It introduces a low-dimensional polynomial basis approach for weight parameterization in neural ODEs and ResNets, enhancing training stability and computational efficiency.

Findings

Legendre basis improves training stability

Reduces computational cost compared to unconstrained models

Achieves accuracy comparable or better than existing methods

Abstract

Continuous-time deep learning models, such as neural ordinary differential equations (ODEs), offer a promising framework for surrogate modeling of complex physical systems. A central challenge in training these models lies in learning expressive yet stable time-varying weights, particularly under computational constraints. This work investigates weight parameterization strategies that constrain the temporal evolution of weights to a low-dimensional subspace spanned by polynomial basis functions. We evaluate both monomial and Legendre polynomial bases within neural ODE and residual network (ResNet) architectures under discretize-then-optimize and optimize-then-discretize training paradigms. Experimental results across three high-dimensional benchmark problems show that Legendre parameterizations yield more stable training dynamics, reduce computational cost, and achieve accuracy comparable to or better than both monomial parameterizations and unconstrained weight models. These findings elucidate the role of basis choice in time-dependent weight parameterization and demonstrate that using orthogonal polynomial bases offers a favorable tradeoff between model expressivity and training efficiency.