libyt: an In Situ Interface Connecting Simulations with yt, Python, and Jupyter Workflows

TL;DR

libyt is an open-source C library that enables in situ analysis and visualization of astrophysical simulation data during runtime, integrating seamlessly with Python, yt, and Jupyter workflows to handle massive datasets efficiently.

Contribution

It introduces libyt, a novel in situ interface that connects simulation codes with Python and yt, facilitating real-time data analysis with minimal workflow disruption.

Findings

libyt enables in situ analysis with minimal overhead.

It supports parallel processing in high-performance computing environments.

Demonstrated integration with GAMER and Enzo for various astrophysical scenarios.

Abstract

In the exascale computing era, handling and analyzing massive datasets have become extremely challenging. In situ analysis, which processes data during simulation runtime and bypasses costly intermediate disk input and output steps, offers a promising solution. We present libyt (https://github.com/yt-project/libyt), an open-source C library that enables astrophysical simulations to analyze and visualize data in parallel computation with yt or other Python packages. libyt can invoke Python routines automatically or provide interactive entry points via a Python prompt or a Jupyter Notebook. It requires minimal intervention in researchers' workflow, allowing users to reuse job submission scripts and Python routines. We describe libyt's architecture for parallel computing in high-performance computing environments, including its bidirectional connection between simulation codes and Python, and its integration into the Jupyter ecosystem. We detail its methods for reading patch-based adaptive mesh refinement (AMR) simulations and handling in-memory data with minimal overhead, and procedures for yielding data when requested by Python. We describe how libyt maps simulation data to yt frontends, allowing post-processing scripts to be converted into in situ analysis with just two lines of change. We document libyt's API and demonstrate its integration into two astrophysical simulation codes, GAMER and Enzo, using examples including core-collapse supernovae, isolated dwarf galaxies, fuzzy dark matter, the Sod shock tube test, Kelvin-Helmholtz instability, and the AGORA galaxy simulation. Finally, we discuss libyt's performance, limitations related to data redistribution, extensibility, architecture, and comparisons with traditional post-processing approaches.