High-Performance I/O: HDF5 for Lattice QCD

TL;DR

This paper presents the integration of parallel HDF5 I/O into lattice QCD software, improving performance and compatibility for high-performance computing applications in quantum field theory simulations.

Contribution

It introduces a new HDF5-based I/O framework for lattice QCD, enhancing performance, reliability, and data structure unification in high-performance computing environments.

Findings

HDF5 implementation outperforms QIO by 10-20%

Room for further performance improvements via dataset chunking

Supports high-performance, reliable, and compatible data management

Abstract

Practitioners of lattice QCD/QFT have been some of the primary pioneer users of the state-of-the-art high-performance-computing systems, and contribute towards the stress tests of such new machines as soon as they become available. As with all aspects of high-performance-computing, I/O is becoming an increasingly specialized component of these systems. In order to take advantage of the latest available high-performance I/O infrastructure, to ensure reliability and backwards compatibility of data files, and to help unify the data structures used in lattice codes, we have incorporated parallel HDF5 I/O into the SciDAC supported USQCD software stack. Here we present the design and implementation of this I/O framework. Our HDF5 implementation outperforms optimized QIO at the 10-20% level and leaves room for further improvement by utilizing appropriate dataset chunking.