Gaia Data Release 2: processing of the photometric data

TL;DR

The Gaia DR2 photometric data processing used a novel Hadoop-based pipeline, achieving high accuracy and a large, well-calibrated catalog of 2.5 billion sources through iterative, data-driven calibration methods.

Contribution

This paper introduces PhotPipe, the first Hadoop-based processing system for a large astrophysical survey, enabling scalable and reliable Gaia photometric data calibration.

Findings

Processed 22 months of Gaia data with 0.9 billion observations per day.

Produced a catalog of 2.5 billion sources with high-quality photometry.

Demonstrated the effectiveness of Hadoop for large-scale astrophysical data processing.

Abstract

The second Gaia data release is based on 22 months of mission data with an average of 0.9 billion individual CCD observations per day. A data volume of this size and granularity requires a robust and reliable but still flexible system to achieve the demanding accuracy and precision constraints that Gaia is capable of delivering. The internal Gaia photometric system was initialised using an iterative process that is solely based on Gaia data. A set of calibrations was derived for the entire Gaia DR2 baseline and then used to produce the final mean source photometry. The photometric catalogue contains 2.5 billion sources comprised of three different grades depending on the availability of colour information and the procedure used to calibrate them: 1.5 billion gold, 144 million silver, and 0.9 billion bronze. These figures reflect the results of the photometric processing; the content of the data release will be different due to the validation and data quality filters applied during the catalogue preparation. The photometric processing pipeline, PhotPipe, implements all the processing and calibration workflows in terms of Map/Reduce jobs based on the Hadoop platform. This is the first example of a processing system for a large astrophysical survey project to make use of these technologies. The improvements in the generation of the integrated G-band fluxes, in the attitude modelling, in the cross-matching, and and in the identification of spurious detections led to a much cleaner input stream for the photometric processing. This, combined with the improvements in the definition of the internal photometric system and calibration flow, produced high-quality photometry. Hadoop proved to be an excellent platform choice for the implementation of PhotPipe in terms of overall performance, scalability, downtime, and manpower required for operations and maintenance.