A Rigorous Uncertainty-Aware Quantification Framework Is Essential for Reproducible and Replicable Machine Learning Workflows

TL;DR

This paper emphasizes the importance of a rigorous uncertainty-aware framework based on Bayesian uncertainty quantification to enhance reproducibility and trustworthiness in scientific machine learning workflows.

Contribution

It introduces a Bayesian uncertainty quantification framework to assess reproducibility in complex scientific ML/AI workflows, addressing a critical gap in current practices.

Findings

Framework enables quantification of prediction variability impact

Enhances trustworthiness of scientific ML/AI results

Supports design of more reproducible workflows

Abstract

The ability to replicate predictions by machine learning (ML) or artificial intelligence (AI) models and results in scientific workflows that incorporate such ML/AI predictions is driven by numerous factors. An uncertainty-aware metric that can quantitatively assess the reproducibility of quantities of interest (QoI) would contribute to the trustworthiness of results obtained from scientific workflows involving ML/AI models. In this article, we discuss how uncertainty quantification (UQ) in a Bayesian paradigm can provide a general and rigorous framework for quantifying reproducibility for complex scientific workflows. Such as framework has the potential to fill a critical gap that currently exists in ML/AI for scientific workflows, as it will enable researchers to determine the impact of ML/AI model prediction variability on the predictive outcomes of ML/AI-powered workflows. We expect that the envisioned framework will contribute to the design of more reproducible and trustworthy workflows for diverse scientific applications, and ultimately, accelerate scientific discoveries.