Minimal Re-computation for Exploratory Data Analysis in Astronomy

TL;DR

This paper introduces a memoization-based technique to minimize re-computation in iterative data analysis workflows, enhancing efficiency, reproducibility, and reducing user errors in radio astronomy data processing.

Contribution

It presents a novel method for automatic re-computation minimization using memoization and referential transparency, with a prototype implementation for CASA.

Findings

Significant reduction in re-computation time during exploratory analysis.

Improved reproducibility of data processing results.

Decreased user errors in iterative data analysis.

Abstract

We present a technique to automatically minimise the re-computation when a data processing program is iteratively changed, or added to, as is often the case in exploratory data analysis in radio astronomy. A typical example is flagging and calibration of demanding or unusual observations where visual inspection suggests improvement to the processing strategy. The technique is based on memoization and referentially transparent tasks. We describe a prototype implementation for the CASA data reduction package. This technique improves the efficiency of data analysis while reducing the possibility for user error and improving the reproducibility of the final result.