Reducing and Analyzing the PHAT Survey with the Cloud

TL;DR

This paper details the development of an efficient cloud-based pipeline for reducing the PHAT survey photometric data, addressing technical challenges and providing publicly accessible data products.

Contribution

It introduces a scalable, cloud-based architecture for photometric data reduction and shares techniques for efficient processing and data dissemination.

Findings

Successful implementation of a cloud-based photometry pipeline

Public availability of processed PHAT data in multiple formats

Insights into optimizing cloud resources for astronomical data reduction

Abstract

We discuss the technical challenges we faced and the techniques we used to overcome them when reducing the PHAT photometric data set on the Amazon Elastic Compute Cloud (EC2). We first describe the architecture of our photometry pipeline, which we found particularly efficient for reducing the data in multiple ways for different purposes. We then describe the features of EC2 that make this architecture both efficient to use and challenging to implement. We describe the techniques we adopted to process our data, and suggest ways these techniques may be improved for those interested in trying such reductions in the future. Finally, we summarize the output photometry data products, which are now hosted publicly in two places in two formats. They are in simple fits tables in the high-level science products on MAST, and on a queryable database available through the NOAO Data Lab.