Towards the processing, review, and delivery of 80% of the ALMA data by the Joint ALMA Observatory (JAO)

TL;DR

This paper details the Joint ALMA Observatory's methodology to process, review, and deliver 80% of ALMA's data within 30 days, significantly improving data management efficiency over previous cycles.

Contribution

It introduces a new methodology enabling the processing of 80% of ALMA data within 30 days, a substantial increase from prior cycles.

Findings

Achieved processing of 80% of data within 30 days

Improved pipeline automation and review processes

Enhanced data quality assurance procedures

Abstract

After eight observing Cycles, the Atacama Large Millimeter-submillimeter Array (ALMA) is capable of observing in eight different bands (covering a frequency range from 84 to 950 GHz), with 66 antennas and two correlators. For the current Cycle (7), ALMA offers up to 4300 hours for the 12-m array, and 3000 hours on both the 7-m of the Atacama Compact Array (ACA) and TP Array plus 750 hours in a supplemental call. From the customer perspective (i.e., the astronomical community), ALMA is an integrated product service provider, i.e. it observes in service mode, processes and delivers the data obtained. The Data Management Group (DMG) is in charge of the processing, reviewing, and delivery of the ALMA data and consists of approximately 60 experts in data reduction, from the ALMA Regional Centers (ARCs) and the Joint ALMA Observatory (JAO), distributed in fourteen countries. Prior to their delivery, the ALMA data products go through a thorough quality assurance (QA) process, so that the astronomers can work on their science without the need of significant additional calibration re-processing. Currently, around 90% of the acquired data is processed with the ALMA pipeline (the so called pipeline-able data), while the remaining 10% is processed completely manually. The Level-1 Key Performance Indicator set by the Observatory to DMG is that 90% of the pipeline-able data sets (i.e. some 80% of the data sets observed during an observing cycle) must be processed, reviewed and delivered within 30 days of data acquisition. This paper describes the methodology followed by the JAO in order to process near 80% of the total data observed during Cycle 7, a giant leap with respect to approximately 30% in Cycle 4 (October 2016 - September 2017).