Performance Comparison of DAOS and Lustre for Object Data Storage Approaches

TL;DR

This paper compares the performance of DAOS and Lustre object storage systems with traditional file systems in HPC environments, analyzing whether storage technology or software interface impacts performance more.

Contribution

It provides an empirical performance comparison between DAOS, Lustre, and standard file systems for object data storage in HPC, highlighting the influence of underlying storage technology versus interface.

Findings

Object storage systems can outperform traditional file systems in HPC scenarios.

Performance is more influenced by storage technology than by software interface.

The study offers insights into optimizing HPC storage architectures.

Abstract

High-performance object stores are an emerging technology which offers an alternative solution in the field of HPC storage, with potential to address long-standing scalability issues in traditional distributed POSIX file systems due to excessive consistency assurance and metadata prescriptiveness. In this paper we assess the performance of storing object-like data within a standard file system, where the configuration and access mechanisms have not been optimised for object access behaviour, and compare with and investigate the benefits of using an object storage system. Whilst this approach is not exploiting the file system in a standard way, this work allows us to investigate whether the underlying storage technology performance is more or less important than the software interface and infrastructure a file system or object store provides.