A numerical study into neural network surrogate model performance for uncertainty propagation

TL;DR

This study evaluates neural network surrogate models' ability to accurately predict the full distribution, especially the tails, of solutions in stochastic boundary value problems, highlighting challenges with outlier samples.

Contribution

It provides a detailed comparison of neural network architectures and loss functions for modeling stochastic solutions, emphasizing tail accuracy and outlier detection methods.

Findings

Fully connected networks with residual loss perform best at extrapolating extreme samples.

Prediction errors are significantly larger at distribution tails, often an order of magnitude higher.

Outlier samples cause large extrapolation errors, necessitating specialized handling.

Abstract

Neural network surrogate models have emerged as a promising approach to model solution fields for a wide variety of boundary value problems encountered in physical modeling. Stochastic problems represent an area of particularly high interest because of the potential to significantly reduce the repeated evaluation of expensive forward models via traditional numerical solvers when conducting parametric analysis. However, many studies found in the literature primarily focus on the ability of neural network surrogate models to represent deterministic samples or mean field solutions and largely overlook surrogate model performance at the tails of the distribution. The present study examines in detail the ability of neural network surrogate models to capture the full distribution of solution fields over the entire probability space, while emphasis is placed at the tails of the distribution. Serving as a canonical problem is the heat conduction equation with a highly stochastic source term, inducing extremely large variation in the thermal solution field. Comparisons are made between a classic feed-forward fully connected network and a Deep Operator Network architecture, using both data-driven and physics-informed loss functions. Results show that the worst-case prediction errors are an order of magnitude larger than the mean field error, highlighting the importance of the outlier samples. The large errors associated with extreme samples result from the networks having to extrapolate beyond the bounds of the training data. A method for identifying these samples is presented along with a discussion of potential approaches to account of their errors. Among the models considered, the fully connected neural network trained using a weak form residual loss performs best in handling these extrapolated inputs, achieving the highest prediction accuracy for the numerically produced datasets.