Using script generators for pipeline prototyping

TL;DR

This paper introduces a three-level software design approach using script generators as prototypes to facilitate the development of automated data calibration and imaging pipelines, demonstrated with ALMA observatory data.

Contribution

It presents a transferable method employing script generators as prototypes within a three-stage software development process for astronomical pipelines.

Findings

Effective for ALMA data calibration and imaging

Supports transition from prototyping to automation

Applicable to various observatory projects

Abstract

Fully automated astronomical data calibration and imaging pipelines are difficult to develop without a good prototyping method which permits to bridge the time between observatory commissioning and the moment when the special features and possible problems of the data and their processing are fully understood. In this paper I present a method which has worked well for the ALMA observatory and which is sufficiently general to be transferable to most other projects. In short, the idea is to use a three-level data analysis software design (scriptable toolkit, script generator, automated pipeline) and a corresponding timing of the software development which is ramping up the effort in three stages starting at the beginning of construction, at the beginning of commissioning, and at the end of commissioning respectively. The important design pattern which I would like to underline here is the use of script generators as prototypes for the automated pipeline.