Three Practical Workflow Schedulers for Easy Maximum Parallelism

TL;DR

This paper introduces three simple, robust workflow schedulers tailored for high-performance computing, demonstrating their scalability and suitability for maximizing system utilization in complex scientific simulations.

Contribution

It presents three new workflow schedulers with simple designs optimized for HPC environments, addressing specific use cases and integrating well with MPI for improved flexibility.

Findings

All three schedulers scale to full Summit utilization.

They have short startup times and low per-task overhead.

The work characterizes minimal task granularity for efficiency.

Abstract

Runtime scheduling and workflow systems are an increasingly popular algorithmic component in HPC because they allow full system utilization with relaxed synchronization requirements. There are so many special-purpose tools for task scheduling, one might wonder why more are needed. Use cases seen on the Summit supercomputer needed better integration with MPI and greater flexibility in job launch configurations. Preparation, execution, and analysis of computational chemistry simulations at the scale of tens of thousands of processors revealed three distinct workflow patterns. A separate job scheduler was implemented for each one using extremely simple and robust designs: file-based, task-list based, and bulk-synchronous. Comparing to existing methods shows unique benefits of this work, including simplicity of design, suitability for HPC centers, short startup time, and well-understood per-task overhead. All three new tools have been shown to scale to full utilization of Summit, and have been made publicly available with tests and documentation. This work presents a complete characterization of the minimum effective task granularity for efficient scheduler usage scenarios. These schedulers have the same bottlenecks, and hence similar task granularities as those reported for existing tools following comparable paradigms.