On instabilities in neural network-based physics simulators

TL;DR

This paper investigates the causes of instability in neural network-based physics simulators, analyzing training dynamics and proposing strategies to improve stability, with implications for both linear and nonlinear systems.

Contribution

The paper provides an analytical understanding of instabilities in neural physics simulators, highlighting the impact of data energy distribution, initialization, and noise injection, and suggests mitigation strategies.

Findings

Training convergence depends on data energy distribution.

Unlearnable directions depend on weight initialization.

Adding synthetic noise can stabilize training but biases dynamics.

Abstract

When neural networks are trained from data to simulate the dynamics of physical systems, they encounter a persistent challenge: the long-time dynamics they produce are often unphysical or unstable. We analyze the origin of such instabilities when learning linear dynamical systems, focusing on the training dynamics. We make several analytical findings which empirical observations suggest extend to nonlinear dynamical systems. First, the rate of convergence of the training dynamics is uneven and depends on the distribution of energy in the data. As a special case, the dynamics in directions where the data have no energy cannot be learned. Second, in the unlearnable directions, the dynamics produced by the neural network depend on the weight initialization, and common weight initialization schemes can produce unstable dynamics. Third, injecting synthetic noise into the data during training adds damping to the training dynamics and can stabilize the learned simulator, though doing so undesirably biases the learned dynamics. For each contributor to instability, we suggest mitigative strategies. We also highlight important differences between learning discrete-time and continuous-time dynamics, and discuss extensions to nonlinear systems.