Quantifying uncertainty for deep learning based forecasting and flow-reconstruction using neural architecture search ensembles

TL;DR

This paper introduces an automated neural architecture search method to create ensembles of deep neural networks for uncertainty quantification in forecasting and flow reconstruction, demonstrating improved performance and reliable uncertainty estimates.

Contribution

It presents a scalable, automated approach combining genetic algorithms and Bayesian optimization to discover neural network ensembles for complex dynamical systems, enabling uncertainty quantification.

Findings

Ensembles outperform individual models and benchmarks.

The method effectively quantifies uncertainty in predictions.

Demonstrated on sea-surface temperature data for forecasting and flow reconstruction.

Abstract

Classical problems in computational physics such as data-driven forecasting and signal reconstruction from sparse sensors have recently seen an explosion in deep neural network (DNN) based algorithmic approaches. However, most DNN models do not provide uncertainty estimates, which are crucial for establishing the trustworthiness of these techniques in downstream decision making tasks and scenarios. In recent years, ensemble-based methods have achieved significant success for the uncertainty quantification in DNNs on a number of benchmark problems. However, their performance on real-world applications remains under-explored. In this work, we present an automated approach to DNN discovery and demonstrate how this may also be utilized for ensemble-based uncertainty quantification. Specifically, we propose the use of a scalable neural and hyperparameter architecture search for discovering an ensemble of DNN models for complex dynamical systems. We highlight how the proposed method not only discovers high-performing neural network ensembles for our tasks, but also quantifies uncertainty seamlessly. This is achieved by using genetic algorithms and Bayesian optimization for sampling the search space of neural network architectures and hyperparameters. Subsequently, a model selection approach is used to identify candidate models for an ensemble set construction. Afterwards, a variance decomposition approach is used to estimate the uncertainty of the predictions from the ensemble. We demonstrate the feasibility of this framework for two tasks - forecasting from historical data and flow reconstruction from sparse sensors for the sea-surface temperature. We demonstrate superior performance from the ensemble in contrast with individual high-performing models and other benchmarks.