Africanus IV. The Stimela2 framework: scalable and reproducible workflows, from local to cloud compute

TL;DR

Stimela2 is a flexible, scalable framework for creating reproducible data reduction workflows in radio astronomy, adaptable to local and cloud computing environments using YAML recipes and containerization.

Contribution

It introduces a modular, YAML-based workflow system that seamlessly integrates containerization and cloud deployment for scalable data processing.

Findings

Supports containerized execution for reproducibility

Enables deployment on cloud and HPC systems

Provides a user-friendly, modular workflow design

Abstract

Stimela2 is a new-generation framework for developing data reduction workflows. It is designed for radio astronomy data but can be adapted for other data processing applications. Stimela2 aims at the middle ground between ease of development, human readability, and enabling robust, scalable and reproducible workflows. It represents workflows by linear, concise and intuitive YAML-format "recipes". Atomic data reduction tasks (binary executables, Python functions and code, and CASA tasks) are described by YAML-format "cab definitions" detailing each task's "schema" (inputs and outputs). Stimela2 provides a rich syntax for chaining tasks together, and encourages a high degree of modularity: recipes may be nested into other recipes, and configuration is cleanly separated from recipe logic. Tasks can be executed natively or in isolated environments using containerization technologies such as Apptainer. The container images are open-source and maintained through a companion package called cult-cargo. This enables the development of system-agnostic and fully reproducible workflows. Stimela2 facilitates the deployment of scalable, distributed workflows by interfacing with the Slurm scheduler and the Kubernetes API. The latter allows workflows to be readily deployed in the cloud. Previous papers in this series used Stimela2 as the underlying technology to run workflows on the AWS cloud. This paper presents an overview of Stimela2's design, architecture and use in the radio astronomy context.