Real-Time Streaming and Event-driven Control of Scientific Experiments

TL;DR

This paper introduces the MDML platform that integrates diverse sensor data streams with machine learning to enable real-time, automated control and optimization of scientific experiments across distributed computing resources.

Contribution

The paper presents the design and implementation of MDML, a platform that standardizes data integration and orchestrates distributed ML for real-time experiment control.

Findings

MDML effectively aggregates IoT data streams for in-situ analysis.

MDML enables real-time decision-making in manufacturing experiments.

The platform orchestrates distributed ML tasks across various computational resources.

Abstract

Advancements in scientific instrument sensors and connected devices provide unprecedented insight into ongoing experiments and present new opportunities for control, optimization, and steering. However, the diversity of sensors and heterogeneity of their data result in make it challenging to fully realize these new opportunities. Organizing and synthesizing diverse data streams in near-real-time requires both rich automation and Machine Learning (ML). To efficiently utilize ML during an experiment, the entire ML lifecycle must be addressed, including refining experiment configurations, retraining models, and applying decisions-tasks that require an equally diverse array of computational resources spanning centralized HPC to the accelerators at the edge. Here we present the Manufacturing Data and Machine Learning platform (MDML). The MDML is designed to standardize the research and operational environment for advanced data analytics and ML-enabled automated process optimization by providing the cyberinfrastructure to integrate sensor data streams and AI in cyber-physical systems for in-situ analysis. To achieve this, the MDML provides a fabric to receive and aggregate IoT data and simultaneously orchestrate remote computation across the computing continuum. In this paper we describe the MDML and show how it is used in advanced manufacturing to act on IoT data and orchestrate distributed ML to guide experiments.