Mitigating Propagation Failures in Physics-informed Neural Networks using Retain-Resample-Release (R3) Sampling

TL;DR

This paper introduces R3 sampling, a novel method to improve physics-informed neural networks by reducing propagation failures, thereby enhancing convergence on complex PDE problems through targeted sampling strategies.

Contribution

The paper proposes the R3 sampling algorithm that adaptively accumulates collocation points in high residual regions, addressing PINN failure modes with minimal computational overhead.

Findings

R3 sampling reduces convergence failures in PINNs.

Empirical results show improved accuracy over baseline methods.

Theoretical analysis supports the effectiveness of R3 sampling.

Abstract

Despite the success of physics-informed neural networks (PINNs) in approximating partial differential equations (PDEs), PINNs can sometimes fail to converge to the correct solution in problems involving complicated PDEs. This is reflected in several recent studies on characterizing the "failure modes" of PINNs, although a thorough understanding of the connection between PINN failure modes and sampling strategies is missing. In this paper, we provide a novel perspective of failure modes of PINNs by hypothesizing that training PINNs relies on successful "propagation" of solution from initial and/or boundary condition points to interior points. We show that PINNs with poor sampling strategies can get stuck at trivial solutions if there are propagation failures, characterized by highly imbalanced PDE residual fields. To mitigate propagation failures, we propose a novel Retain-Resample-Release sampling (R3) algorithm that can incrementally accumulate collocation points in regions of high PDE residuals with little to no computational overhead. We provide an extension of R3 sampling to respect the principle of causality while solving time-dependent PDEs. We theoretically analyze the behavior of R3 sampling and empirically demonstrate its efficacy and efficiency in comparison with baselines on a variety of PDE problems.