A Resourceful Coordination Approach for Multilevel Scheduling

TL;DR

This paper introduces a coordination method that improves high-performance computing system efficiency by enabling schedulers to share idle resources, resulting in up to 12.6% better utilization and reduced execution times.

Contribution

It proposes a novel resourceful coordination approach (RCA) that facilitates cooperation between batch and application schedulers to enhance system utilization.

Findings

System utilization increased by up to 12.6%.

System makespan decreased by up to 12.6%.

Applications' performance remained unaffected.

Abstract

HPC users aim to improve their execution times without particular regard for increasing system utilization. On the contrary, HPC operators favor increasing the number of executed applications per time unit and increasing system utilization. This difference in the preferences promotes the following operational model. Applications execute on exclusively-allocated computing resources for a specific time and applications are assumed to utilize the allocated resources efficiently. In many cases, this operational model is inefficient, i.e., applications may not fully utilize their allocated resources. This inefficiency results in increasing application execution time and decreasing system utilization. In this work, we propose a resourceful coordination approach (RCA) that enables the cooperation between, currently independent, batch- and application-level schedulers. RCA enables application schedulers to share their allocated but idle computing resources with other applications through the batch system. The effective system performance (ESP) benchmark is used to assess the proposed approach. The results show that RCA increased system utilization up to 12.6% and decreased system makespan by the same percent without affecting applications' performance.