Exploring Scientific Application Performance Using Large Scale Object Storage

TL;DR

This paper investigates how large-scale object storage can improve the performance and scalability of scientific applications on supercomputers by emulating its use and evaluating potential benefits.

Contribution

It provides an emulation-based analysis of object storage in scientific applications, highlighting its potential to enhance scalability and performance.

Findings

Object storage can improve I/O performance in scientific applications.

Emulation shows potential scalability benefits of object storage.

Results suggest compatibility of object storage with HPC workloads.

Abstract

One of the major performance and scalability bottlenecks in large scientific applications is parallel reading and writing to supercomputer I/O systems. The usage of parallel file systems and consistency requirements of POSIX, that all the traditional HPC parallel I/O interfaces adhere to, pose limitations to the scalability of scientific applications. Object storage is a widely used storage technology in cloud computing and is more frequently proposed for HPC workload to address and improve the current scalability and performance of I/O in scientific applications. While object storage is a promising technology, it is still unclear how scientific applications will use object storage and what the main performance benefits will be. This work addresses these questions, by emulating an object storage used by a traditional scientific application and evaluating potential performance benefits. We show that scientific applications can benefit from the usage of object storage on large scales.