Accelerating scientific discovery with the common task framework

TL;DR

The paper introduces a common task framework (CTF) to standardize evaluation metrics and datasets, accelerating the development and comparison of ML/AI algorithms across various scientific and engineering disciplines.

Contribution

It presents a new common task framework with challenge datasets for diverse scientific objectives, facilitating fair comparison and advancement of ML/AI methods.

Findings

CTF enables standardized benchmarking of algorithms.

It accelerates progress in ML/AI for science and engineering.

The framework supports diverse objectives like forecasting and control.

Abstract

Machine learning (ML) and artificial intelligence (AI) algorithms are transforming and empowering the characterization and control of dynamic systems in the engineering, physical, and biological sciences. These emerging modeling paradigms require comparative metrics to evaluate a diverse set of scientific objectives, including forecasting, state reconstruction, generalization, and control, while also considering limited data scenarios and noisy measurements. We introduce a common task framework (CTF) for science and engineering, which features a growing collection of challenge data sets with a diverse set of practical and common objectives. The CTF is a critically enabling technology that has contributed to the rapid advance of ML/AI algorithms in traditional applications such as speech recognition, language processing, and computer vision. There is a critical need for the objective metrics of a CTF to compare the diverse algorithms being rapidly developed and deployed in practice today across science and engineering.