Co-scheduling Ensembles of In Situ Workflows

TL;DR

This paper introduces a novel co-scheduling method for in situ workflows of molecular dynamics simulations, significantly reducing total execution time by optimizing resource allocation and management.

Contribution

It presents a new theoretical framework and an efficient scheduling approach for co-scheduling ensembles of simulations and in situ analyses under resource constraints.

Findings

Co-scheduling improves workflow makespan by up to 30 times.

Data locality benefits from coupled simulation and analysis scheduling.

The approach is validated using the WRENCH simulation framework.

Abstract

Molecular dynamics (MD) simulations are widely used to study large-scale molecular systems. HPC systems are ideal platforms to run these studies, however, reaching the necessary simulation timescale to detect rare processes is challenging, even with modern supercomputers. To overcome the timescale limitation, the simulation of a long MD trajectory is replaced by multiple short-range simulations that are executed simultaneously in an ensemble of simulations. Analyses are usually co-scheduled with these simulations to efficiently process large volumes of data generated by the simulations at runtime, thanks to in situ techniques. Executing a workflow ensemble of simulations and their in situ analyses requires efficient co-scheduling strategies and sophisticated management of computational resources so that they are not slowing down each other. In this paper, we propose an efficient method to co-schedule simulations and in situ analyses such that the makespan of the workflow ensemble is minimized. We present a novel approach to allocate resources for a workflow ensemble under resource constraints by using a theoretical framework modeling the workflow ensemble's execution. We evaluate the proposed approach using an accurate simulator based on the WRENCH simulation framework on various workflow ensemble configurations. Results demonstrate the significance of co-scheduling simulations and in situ analyses that couple data together to benefit from data locality, in which inefficient scheduling decisions can lead up to a factor 30 slowdown in makespan.