Convergence of uncertainty estimates in Ensemble and Bayesian sparse model discovery

TL;DR

This paper analyzes ensemble sparse model discovery, demonstrating its theoretical convergence, robustness, and efficiency in uncertainty estimation, especially in low-data and high-noise scenarios, outperforming some existing methods.

Contribution

It provides a theoretical analysis of bootstrapping-based ensemble sparse model discovery, showing exponential convergence and efficient uncertainty quantification compared to Bayesian methods.

Findings

Proves exponential convergence rate of the error in variable selection.

Shows ensemble method's superior performance over LASSO and thresholding least-squares.

Demonstrates valid uncertainty quantification converging with increased sample size.

Abstract

Sparse model identification enables nonlinear dynamical system discovery from data. However, the control of false discoveries for sparse model identification is challenging, especially in the low-data and high-noise limit. In this paper, we perform a theoretical study on ensemble sparse model discovery, which shows empirical success in terms of accuracy and robustness to noise. In particular, we analyse the bootstrapping-based sequential thresholding least-squares estimator. We show that this bootstrapping-based ensembling technique can perform a provably correct variable selection procedure with an exponential convergence rate of the error rate. In addition, we show that the ensemble sparse model discovery method can perform computationally efficient uncertainty estimation, compared to expensive Bayesian uncertainty quantification methods via MCMC. We demonstrate the convergence properties and connection to uncertainty quantification in various numerical studies on synthetic sparse linear regression and sparse model discovery. The experiments on sparse linear regression support that the bootstrapping-based sequential thresholding least-squares method has better performance for sparse variable selection compared to LASSO, thresholding least-squares, and bootstrapping-based LASSO. In the sparse model discovery experiment, we show that the bootstrapping-based sequential thresholding least-squares method can provide valid uncertainty quantification, converging to a delta measure centered around the true value with increased sample sizes. Finally, we highlight the improved robustness to hyperparameter selection under shifting noise and sparsity levels of the bootstrapping-based sequential thresholding least-squares method compared to other sparse regression methods.