Human-in-the-loop: The future of Machine Learning in Automated Electron Microscopy

TL;DR

This paper discusses the integration of human-in-the-loop machine learning for real-time control and decision-making in automated electron microscopy experiments, highlighting its potential and current limitations.

Contribution

It introduces the concept of human-in-the-loop active experiments in electron microscopy and explores design considerations for implementing ML-based real-time control.

Findings

ML is increasingly used for data analysis in electron microscopy.

APIs enable ML deployment in microscopes for real-time feedback.

Human-in-the-loop approach allows human oversight and policy tuning in automated experiments.

Abstract

Machine learning methods are progressively gaining acceptance in the electron microscopy community for de-noising, semantic segmentation, and dimensionality reduction of data post-acquisition. The introduction of the APIs by major instrument manufacturers now allows the deployment of ML workflows in microscopes, not only for data analytics but also for real-time decision-making and feedback for microscope operation. However, the number of use cases for real-time ML remains remarkably small. Here, we discuss some considerations in designing ML-based active experiments and pose that the likely strategy for the next several years will be human-in-the-loop automated experiments (hAE). In this paradigm, the ML learning agent directly controls beam position and image and spectroscopy acquisition functions, and human operator monitors experiment progression in real- and feature space of the system and tunes the policies of the ML agent to steer the experiment towards specific objectives.