Streaming Large-Scale Electron Microscopy Data to a Supercomputing Facility

TL;DR

This paper presents a high-speed streaming workflow that transfers large-scale electron microscopy data directly from acquisition to supercomputing resources, significantly reducing data transfer time and enabling real-time analysis.

Contribution

It introduces a novel streaming pipeline that bypasses intermediate storage, demonstrating its effectiveness in handling 10 TB datasets in a production environment.

Findings

Streaming workflow reduces data transfer time substantially.

Enables real-time data analysis and decision-making.

Minimizes human error by automating data transfer process.

Abstract

Data management is a critical component of modern experimental workflows. As data generation rates increase, transferring data from acquisition servers to processing servers via conventional file-based methods is becoming increasingly impractical. The 4D Camera at the National Center for Electron Microscopy (NCEM) generates data at a nominal rate of 480 Gbit/s (87,000 frames/s) producing a 700 GB dataset in fifteen seconds. To address the challenges associated with storing and processing such quantities of data, we developed a streaming workflow that utilizes a high-speed network to connect the 4D Camera's data acquisition (DAQ) system to supercomputing nodes at the National Energy Research Scientific Computing Center (NERSC), bypassing intermediate file storage entirely. In this work, we demonstrate the effectiveness of our streaming pipeline in a production setting through an hour-long experiment that generated over 10 TB of raw data, yielding high-quality datasets suitable for advanced analyses. Additionally, we compare the efficacy of this streaming workflow against the conventional file-transfer workflow by conducting a post-mortem analysis on historical data from experiments performed by real users. Our findings show that the streaming workflow significantly improves data turnaround time, enables real-time decision-making, and minimizes the potential for human error by eliminating manual user interactions.