XRISM Pre-Pipeline and Singularity: Container-Based Data Processing for the X-Ray Imaging and Spectroscopy Mission and High-Performance Computing

TL;DR

This paper presents a container-based data processing system for the XRISM X-ray observatory, achieving significant speedups using Singularity containers on high-performance computing infrastructure.

Contribution

It introduces a containerized pre-pipeline system utilizing Singularity for XRISM data processing, enhancing performance on HPC systems.

Findings

33x speedup in data processing tasks

Successful porting of XRISM pre-pipeline to HPC environment

Effective use of Singularity containers on high-performance systems

Abstract

The X-Ray Imaging and Spectroscopy Mission (XRISM) is the seventh Japanese X-ray observatory whose development and operation are in collaboration with universities and research institutes in Japan, the United States, and Europe, including JAXA, NASA, and ESA. The telemetry data downlinked from the satellite are reduced to scientific products using pre-pipeline (PPL) and pipeline (PL) software running on standard Linux virtual machines (VMs) for the JAXA and NASA sides, respectively. OBSIDs identified the observations, and we had 80 and 161 OBSIDs to be reprocessed at the end of the commissioning period and performance verification and calibration period, respectively. The combination of the containerized PPL utilizing Singularity of a container platform running on the JAXA's "TOKI-RURI" high-performance computing (HPC) system and working disk images formatted to ext3 accomplished a 33x speedup in PPL tasks over our regular VM. Herein, we briefly describe the data processing in XRISM and our porting strategies for PPL in the HPC environment.