AdapTBF: Decentralized Bandwidth Control via Adaptive Token Borrowing for HPC Storage

TL;DR

AdapTBF is a decentralized bandwidth control mechanism for HPC storage that adaptively manages I/O bandwidth through token borrowing, improving efficiency and fairness over traditional proportional limits.

Contribution

It introduces a novel adaptive token borrowing algorithm for decentralized bandwidth control in parallel file systems like Lustre.

Findings

AdapTBF improves storage utilization under bursty workloads.

It maintains fairness among applications with diverse I/O patterns.

AdapTBF outperforms strict proportional limits in efficiency and fairness.

Abstract

Modern high-performance computing (HPC) applications run on compute resources but share global storage systems. This design can cause problems when applications consume a disproportionate amount of storage bandwidth relative to their allocated compute resources. For example, an application running on a single compute node can issue many small, random writes and consume excessive I/O bandwidth from a storage server. This can hinder larger jobs that write to the same storage server and are allocated many compute nodes, resulting in significant resource waste. A straightforward solution is to limit each application's I/O bandwidth on storage servers in proportion to its allocated compute resources. This approach has been implemented in parallel file systems using Token Bucket Filter (TBF). However, strict proportional limits often reduce overall I/O efficiency because HPC applications generate short, bursty I/O. Limiting bandwidth can waste server capacity when applications are idle or prevent applications from temporarily using higher bandwidth during bursty phases. We argue that I/O control should maximize per-application performance and overall storage efficiency while ensuring fairness (e.g., preventing small jobs from blocking large-scale ones). We propose AdapTBF, which builds on TBF in modern parallel file systems (e.g., Lustre) and introduces a decentralized bandwidth control approach using adaptive borrowing and lending. We detail the algorithm, implement AdapTBF in Lustre, and evaluate it using synthetic workloads modeled after real-world scenarios. Results show that AdapTBF manages I/O bandwidth effectively while maintaining high storage utilization, even under extreme conditions.