Adaptive parallelism with RMI: Idle high-performance computing resources can be completely avoided

TL;DR

This paper presents an adaptive parallelism approach using RMI that dynamically utilizes idle high-performance computing resources, significantly improving hardware utilization without pre-set resource constraints.

Contribution

It introduces a novel adaptive parallelism method leveraging RMI to fully utilize idle HPC resources, eliminating the need for fixed resource requests.

Findings

Achieves near-complete hardware utilization in real-world supercomputing environments.

Demonstrates effectiveness of adaptive parallelism in filling scheduling gaps.

Reduces idle time of high-performance computing resources.

Abstract

In practice, standard scheduling of parallel computing jobs almost always leaves significant portions of the available hardware unused, even with many jobs still waiting in the queue. The simple reason is that the resource requests of these waiting jobs are fixed and do not match the available, unused resources. However, with alternative but existing and well-established techniques it is possible to achieve a fully automated, adaptive parallelism that does not need pre-set, fixed resources. Here, we demonstrate that such an adaptively parallel program can indeed fill in all such scheduling gaps, even in real-life situations on large supercomputers.