Probabilistic Scheduling of Dynamic I/O Requests via Application Clustering for Burst-Buffer Equipped HPC

TL;DR

This paper introduces a dynamic probabilistic I/O scheduling framework for burst-buffer equipped HPC systems, using application clustering to adapt to changing workloads and improve I/O performance.

Contribution

It proposes a novel application clustering-based framework (DPSAC) that dynamically adjusts probabilistic I/O scheduling to handle application variability in HPC environments.

Findings

Effective in reducing I/O congestion in simulations

Adapts to dynamic application join and exit scenarios

Improves load balancing across burst buffers

Abstract

Burst-Buffering is a promising storage solution that introduces an intermediate highthroughput storage buffer layer to mitigate the I/O bottleneck problem that the current High-Performance Computing (HPC) platforms suffer. The existing Markov-Chain based probabilistic I/O scheduling utilizes the load state of Burst-Buffers and the periodical characteristics of applications to reduce I/O congestion due to the limited capacity of Burst-Buffers. However, this probabilistic approach requires consistent I/O characteristics of applications, including similar I/O duration and long application length, in order to obtain an accurate I/O load estimation. These consistency conditions do not often hold in realistic situations. In this paper, we propose a generic framework of dynamic probabilistic I/O scheduling based on application clustering (DPSAC) to make applications meet the consistency requirements. According to the I/O phrase length of each application, our scheme first deploys a one-dimensional K-means clustering algorithm to cluster the applications into clusters. Next, it calculates the expected workload of each cluster through the probabilistic model of applications and then partitions the Burst-Buffers proportionally. Then, to handle dynamic changes (join and exit) of applications, it updates the clusters based on a heuristic strategy. Finally, it applies the probabilistic I/O scheduling, which is based on the distribution of application workload and the state of Burst-Buffers, to schedule I/O for all the concurrent applications to mitigate I/O congestion. The simulation results on synthetic data show that our DPSAC is effective and efficient.