The Astro-WISE Optical Image Pipeline: Development and Implementation

TL;DR

The paper presents Astro-WISE, a comprehensive, integrated system for processing, calibrating, and analyzing wide-field astronomical data across distributed hardware, enabling complete data lineage and efficient handling of large datasets.

Contribution

It introduces a novel, fully integrated pipeline system that links raw data to final catalogs within a distributed environment, enhancing large-scale astronomical data processing.

Findings

Successful processing of data from WFI and OmegaCAM instruments

Complete data lineage from raw to final products achieved

Improved efficiency in survey management and quality control

Abstract

We have designed and implemented a novel way to process wide-field astronomical data within a distributed environment of hardware resources and humanpower. The system is characterized by integration of archiving, calibration, and post-calibration analysis of data from raw, through intermediate, to final data products. It is a true integration thanks to complete linking of data lineage from the final catalogs back to the raw data. This paper describes the pipeline processing of optical wide-field astronomical data from the WFI (http://www.eso.org/lasilla/instruments/wfi/) and OmegaCAM (http://www.astro-wise.org/~omegacam/) instruments using the Astro-WISE information system (the Astro-WISE Environment or simply AWE). This information system is an environment of hardware resources and humanpower distributed over Europe. AWE is characterized by integration of archiving, data calibration, post-calibration analysis, and archiving of raw, intermediate, and final data products. The true integration enables a complete data processing cycle from the raw data up to the publication of science-ready catalogs. The advantages of this system for very large datasets are in the areas of: survey operations management, quality control, calibration analyses, and massive processing.