PADLL: Taming Metadata-intensive HPC Jobs Through Dynamic, Application-agnostic QoS Control

TL;DR

PADLL is a middleware solution that manages data and metadata workflows in HPC storage systems, ensuring QoS, fairness, and prioritization for metadata-intensive jobs through dynamic, application-agnostic control.

Contribution

It introduces PADLL, a novel middleware that applies Software-Defined Storage principles to enforce QoS policies in HPC environments, handling metadata and data workflows effectively.

Findings

Enforces complex QoS policies for concurrent jobs.

Ensures fairness and prioritization in metadata operations.

Proven effective with synthetic benchmarks, real applications, and production traces.

Abstract

Modern I/O applications that run on HPC infrastructures are increasingly becoming read and metadata intensive. However, having multiple concurrent applications submitting large amounts of metadata operations can easily saturate the shared parallel file system's metadata resources, leading to overall performance degradation and I/O unfairness. We present PADLL, an application and file system agnostic storage middleware that enables QoS control of data and metadata workflows in HPC storage systems. It adopts ideas from Software-Defined Storage, building data plane stages that mediate and rate limit POSIX requests submitted to the shared file system, and a control plane that holistically coordinates how all I/O workflows are handled. We demonstrate its performance and feasibility under multiple QoS policies using synthetic benchmarks, real-world applications, and traces collected from a production file system. Results show that PADLL can enforce complex storage QoS policies over concurrent metadata-aggressive jobs, ensuring fairness and prioritization.