Improving the performance of Stein variational inference through extreme sparsification of physically-constrained neural network models

TL;DR

This paper introduces an $L_0$ sparsification prior combined with Stein variational gradient descent to improve uncertainty quantification in neural network models for scientific machine learning, reducing computational costs and enhancing robustness.

Contribution

The novel integration of $L_0$ sparsification with SVGD offers a more robust and efficient approach for uncertainty quantification in high-dimensional neural network models.

Findings

$L_0$+SVGD outperforms standard SVGD in noise resilience.

$L_0$+SVGD achieves faster convergence to optimal solutions.

$L_0$+SVGD performs well in extrapolated regions.

Abstract

Most scientific machine learning (SciML) applications of neural networks involve hundreds to thousands of parameters, and hence, uncertainty quantification for such models is plagued by the curse of dimensionality. Using physical applications, we show that $L_0$ sparsification prior to Stein variational gradient descent ($L_0$+SVGD) is a more robust and efficient means of uncertainty quantification, in terms of computational cost and performance than the direct application of SGVD or projected SGVD methods. Specifically, $L_0$+SVGD demonstrates superior resilience to noise, the ability to perform well in extrapolated regions, and a faster convergence rate to an optimal solution.