Mitigating Shared Storage Congestion Using Control Theory

TL;DR

This paper introduces a control theory-based method to dynamically regulate I/O rates in shared HPC environments, effectively reducing congestion and improving overall performance stability.

Contribution

It presents a novel self-adaptive control approach that uses runtime metrics to mitigate shared storage congestion in HPC systems.

Findings

Reduces total runtime by up to 20%.

Lowers tail latency under workload.

Maintains stable performance during congestion.

Abstract

Efficient data access in High-Performance Computing (HPC) systems is essential to the performance of intensive computing tasks. Traditional optimizations of the I/O stack aim to improve peak performance but are often workload specific and require deep expertise, making them difficult to generalize or re-use. In shared HPC environments, resource congestion can lead to unpredictable performance, causing slowdowns and timeouts. To address these challenges, we propose a self-adaptive approach based on Control Theory to dynamically regulate client-side I/O rates. Our approach leverages a small set of runtime system load metrics to reduce congestion and enhance performance stability. We implement a controller in a multi-node cluster and evaluate it on a real testbed under a representative workload. Experimental results demonstrate that our method effectively mitigates I/O congestion, reducing total runtime by up to 20% and lowering tail latency, while maintaining stable performance.