ElasticBroker: Combining HPC with Cloud to Provide Realtime Insights into Simulations

TL;DR

ElasticBroker enables real-time data streaming and analysis by integrating HPC simulations with Cloud-based processing, reducing I/O costs and improving scalability for scientific workflows.

Contribution

It introduces ElasticBroker, a framework that bridges HPC and Cloud systems for in-situ analysis, facilitating data transformation and streaming between the two ecosystems.

Findings

Achieves high-quality, scalable real-time analysis of simulations.

Supports decoupled HPC and Cloud workflows with good performance.

Enables continuous data streaming from simulations to Cloud analytics.

Abstract

For large-scale scientific simulations, it is expensive to store raw simulation results to perform post-analysis. To minimize expensive I/O, "in-situ" analysis is often used, where analysis applications are tightly coupled with scientific simulations and can access and process the simulation results in memory. Increasingly, scientific domains employ Big Data approaches to analyze simulations for scientific discoveries. However, it remains a challenge to organize, transform, and transport data at scale between the two semantically different ecosystems (HPC and Cloud systems). In an effort to address these challenges, we design and implement the ElasticBroker software framework, which bridges HPC and Cloud applications to form an "in-situ" scientific workflow. Instead of writing simulation results to parallel file systems, ElasticBroker performs data filtering, aggregation, and format conversions to close the gap between an HPC ecosystem and a distinct Cloud ecosystem. To achieve this goal, ElasticBroker reorganizes simulation snapshots into continuous data streams and send them to the Cloud. In the Cloud, we deploy a distributed stream processing service to perform online data analysis. In our experiments, we use ElasticBroker to setup and execute a cross-ecosystem scientific workflow, which consists of a parallel computational fluid dynamics (CFD) simulation running on a supercomputer, and a parallel dynamic mode decomposition (DMD) analysis application running in a Cloud computing platform. Our results show that running scientific workflows consisting of decoupled HPC and Big Data jobs in their native environments with ElasticBroker, can achieve high quality of service, good scalability, and provide high-quality analytics for ongoing simulations.