Astronomical Image Processing at Scale With Pegasus and Montage

TL;DR

This paper demonstrates how the Pegasus Workflow Manager Python API, combined with the Montage Image Mosaic engine, efficiently manages large-scale astronomical image processing workflows, enabling automation, scalability, and performance optimization.

Contribution

It showcases the application of Pegasus and Montage for scalable, automated astronomical image mosaicking, highlighting their effectiveness across various scientific projects.

Findings

Pegasus effectively manages large-scale image processing workflows.

Montage enables efficient creation of astronomical mosaics.

Workflow automation improves reliability and performance in data analysis.

Abstract

Image processing at scale is a powerful tool for creating new data sets and integrating them with existing data sets and performing analysis and quality assurance investigations. Workflow managers offer advantages in this type of processing, which involves multiple data access and processing steps. Generally, they enable automation of the workflow by locating data and resources, recovery from failures, and monitoring of performance. In this focus demo we demonstrate how the Pegasus Workflow Manager Python API manages image processing to create mosaics with the Montage Image Mosaic engine. Since 2001, Pegasus has been developed and maintained at USC/ISI. Montage was in fact one of the first applications used to design Pegasus and optimize its performance. Pegasus has since found application in many areas of science. LIGO exploited it in making discoveries of black holes. The Vera C. Rubin Observatory used it to compare the cost and performance of processing images on cloud platforms. While these are examples of projects at large scale, small team investigations on local clusters of machines can benefit from Pegasus as well.