Scientific Machine Learning Benchmarks

TL;DR

This paper discusses the development of benchmarks for scientific machine learning to evaluate and compare different algorithms, frameworks, and architectures in analyzing large scientific datasets generated by modern experimental facilities.

Contribution

It introduces a new benchmarking approach tailored for scientific machine learning and reviews existing methods, addressing the challenge of selecting appropriate algorithms for large datasets.

Findings

Proposes a framework for benchmarking scientific machine learning methods.

Reviews existing benchmarking approaches in scientific machine learning.

Highlights the importance of metrics for evaluating machine learning in scientific contexts.

Abstract

The breakthrough in Deep Learning neural networks has transformed the use of AI and machine learning technologies for the analysis of very large experimental datasets. These datasets are typically generated by large-scale experimental facilities at national laboratories. In the context of science, scientific machine learning focuses on training machines to identify patterns, trends, and anomalies to extract meaningful scientific insights from such datasets. With a new generation of experimental facilities, the rate of data generation and the scale of data volumes will increasingly require the use of more automated data analysis. At present, identifying the most appropriate machine learning algorithm for the analysis of any given scientific dataset is still a challenge for scientists. This is due to many different machine learning frameworks, computer architectures, and machine learning models. Historically, for modelling and simulation on HPC systems such problems have been addressed through benchmarking computer applications, algorithms, and architectures. Extending such a benchmarking approach and identifying metrics for the application of machine learning methods to scientific datasets is a new challenge for both scientists and computer scientists. In this paper, we describe our approach to the development of scientific machine learning benchmarks and review other approaches to benchmarking scientific machine learning.