Lemon: an MPI parallel I/O library for data encapsulation using LIME

TL;DR

Lemon is an MPI parallel I/O library designed for efficient data and metadata storage in Lattice QCD applications, demonstrating excellent scalability on high-performance computing systems.

Contribution

It introduces Lemon, a novel MPI parallel I/O library that supports Lattice QCD data format and scalable performance on supercomputers.

Findings

Excellent scaling properties demonstrated on high-performance computers.

Supports large binary data and metadata in a unified format.

Designed for Lattice QCD but applicable to similar data profiles.

Abstract

We introduce Lemon, an MPI parallel I/O library that is intended to allow for efficient parallel I/O of both binary and metadata on massively parallel architectures. Motivated by the demands of the Lattice Quantum Chromodynamics community, the data is stored in the SciDAC Lattice QCD Interchange Message Encapsulation format. This format allows for storing large blocks of binary data and corresponding metadata in the same file. Even if designed for LQCD needs, this format might be useful for any application with this type of data profile. The design, implementation and application of Lemon are described. We conclude with presenting the excellent scaling properties of Lemon on state of the art high performance computers.