Development of a Burst Buffer System for Data-Intensive Applications

TL;DR

This paper presents the development of a burst buffer system designed to improve I/O performance for data-intensive scientific applications by buffering data temporarily to handle bursty I/O patterns more efficiently.

Contribution

It introduces a burst buffer system layered on Lustre to address I/O bottlenecks caused by bursty data traffic in scientific applications.

Findings

Burst buffer system improves I/O handling during checkpointing.

Utilizing burst buffers reduces I/O bottlenecks in Lustre.

Initial results show promising performance gains.

Abstract

Modern parallel filesystems such as Lustre are designed to provide high, scalable I/O bandwidth in response to growing I/O requirements; however, the bursty I/O characteristics of many data-intensive scientific applications make it difficult for back-end parallel filesystems to efficiently handle I/O requests. A burst buffer system, through which data can be temporarily buffered via high-performance storage mediums, allows for gradual flushing of data to back-end filesystems. In this paper, we explore issues surrounding the development of a burst buffer system for data-intensive scientific applications. Our initial results demonstrate that utilizing a burst buffer system on top of the Lustre filesystem shows promise for dealing with the intense I/O traffic generated by application checkpointing.