ElasticBroker: Combining HPC with Cloud to Provide Realtime Insights into Simulations
Feng Li, Dali Wang, Feng Yan, Fengguang Song

TL;DR
ElasticBroker enables real-time data streaming and analysis by integrating HPC simulations with Cloud-based processing, reducing I/O costs and improving scalability for scientific workflows.
Contribution
It introduces ElasticBroker, a framework that bridges HPC and Cloud systems for in-situ analysis, facilitating data transformation and streaming between the two ecosystems.
Findings
Achieves high-quality, scalable real-time analysis of simulations.
Supports decoupled HPC and Cloud workflows with good performance.
Enables continuous data streaming from simulations to Cloud analytics.
Abstract
For large-scale scientific simulations, it is expensive to store raw simulation results to perform post-analysis. To minimize expensive I/O, "in-situ" analysis is often used, where analysis applications are tightly coupled with scientific simulations and can access and process the simulation results in memory. Increasingly, scientific domains employ Big Data approaches to analyze simulations for scientific discoveries. However, it remains a challenge to organize, transform, and transport data at scale between the two semantically different ecosystems (HPC and Cloud systems). In an effort to address these challenges, we design and implement the ElasticBroker software framework, which bridges HPC and Cloud applications to form an "in-situ" scientific workflow. Instead of writing simulation results to parallel file systems, ElasticBroker performs data filtering, aggregation, and format…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsScientific Computing and Data Management · Advanced Data Storage Technologies · Distributed and Parallel Computing Systems
