XPipeline: Starlight subtraction at scale for MagAO-X

TL;DR

XPipeline is a scalable data processing pipeline designed for high-volume adaptive optics data from MagAO-X, enabling efficient starlight subtraction and analysis at various computational scales.

Contribution

The paper introduces a flexible, scalable pipeline architecture for processing large-scale ExAO data, adaptable from laptops to HPC clusters.

Findings

Pipeline effectively handles 1 TB/hr data rates.

Supports both real-time and post-processing workflows.

Demonstrated on recent direct-imaging data.

Abstract

MagAO-X is an extreme adaptive optics (ExAO) instrument for the Magellan Clay 6.5-meter telescope at Las Campanas Observatory in Chile. Its high spatial and temporal resolution can produce data rates of 1 TB/hr or more, including all AO system telemetry and science images. We describe the tools and architecture we use for commanding, telemetry, and science data transmission and storage. The high data volumes require a distributed approach to data processing, and we have developed a pipeline that can scale from a single laptop to dozens of HPC nodes. The same codebase can then be used for both quick-look functionality at the telescope and for post-processing. We present the software and infrastructure we have developed for ExAO data post-processing, and illustrate their use with recently acquired direct-imaging data.