Node-Based Job Scheduling for Large Scale Simulations of Short Running Jobs

TL;DR

This paper introduces a node-based scheduling method for large-scale simulations that significantly improves scheduler performance, enabling efficient resource utilization for both short and long jobs on supercomputing systems.

Contribution

The paper presents a novel node-based scheduling approach that achieves up to 100 times faster performance than existing systems for large-scale short jobs.

Findings

Up to 100x faster scheduler performance.

Efficient resource utilization for mixed workloads.

Applicable to large-scale supercomputing environments.

Abstract

Diverse workloads such as interactive supercomputing, big data analysis, and large-scale AI algorithm development, requires a high-performance scheduler. This paper presents a novel node-based scheduling approach for large scale simulations of short running jobs on MIT SuperCloud systems, that allows the resources to be fully utilized for both long running batch jobs while simultaneously providing fast launch and release of large-scale short running jobs. The node-based scheduling approach has demonstrated up to 100 times faster scheduler performance that other state-of-the-art systems.