Evaluating Dynamic File Striping For Lustre

TL;DR

This paper evaluates dynamic file striping in Lustre, where striping parameters change as files grow, showing benefits for large sequential read workloads but mixed results for random access patterns.

Contribution

It introduces a watermark-based strategy for dynamic striping in Lustre and assesses its performance across various workloads.

Findings

Improves performance for large sequential reads

Less effective for workloads with random reads

Dynamic striping benefits depend on workload characteristics

Abstract

We define dynamic striping as the ability to assign different Lustre striping characteristics to contiguous segments of a file as it grows. In this paper, we evaluate the effects of dynamic striping using a watermark-based strategy where the stripe count or width is increased once a file's size exceeds one of the chosen watermarks. To measure the performance of this strategy we used a modified version of the IOR benchmark, a netflow analysis workload, and the blastn algorithm from NCBI BLAST. The results indicate that dynamic striping is beneficial to tasks with unpredictable data file size and large sequential reads, but are less conclusive for workloads with significant random read phases.