Sequential-in-time training of nonlinear parametrizations for solving time-dependent partial differential equations

TL;DR

This paper unifies sequential-in-time training methods for nonlinear PDE solutions with classical numerical analysis schemes, providing new stability, error analysis, and insights into over-fitting phenomena.

Contribution

It offers a unifying framework connecting OtD and DtO schemes, leading to novel stability and error results, and links gradient descent methods to classical schemes.

Findings

Unified perspective on OtD and DtO schemes.

New stability and a posteriori error analysis results.

Identification of gradient descent as OtD applied to gradient flows.

Abstract

Sequential-in-time methods solve a sequence of training problems to fit nonlinear parametrizations such as neural networks to approximate solution trajectories of partial differential equations over time. This work shows that sequential-in-time training methods can be understood broadly as either optimize-then-discretize (OtD) or discretize-then-optimize (DtO) schemes, which are well known concepts in numerical analysis. The unifying perspective leads to novel stability and a posteriori error analysis results that provide insights into theoretical and numerical aspects that are inherent to either OtD or DtO schemes such as the tangent space collapse phenomenon, which is a form of over-fitting. Additionally, the unified perspective facilitates establishing connections between variants of sequential-in-time training methods, which is demonstrated by identifying natural gradient descent methods on energy functionals as OtD schemes applied to the corresponding gradient flows.