Automated curation of infra-red imaging data in the WFCAM and VISTA Science Archives

TL;DR

This paper discusses the automation of data curation processes for infrared imaging data from UKIRT-WFCAM and VISTA surveys, aiming to handle vast datasets efficiently with minimal manual intervention.

Contribution

It presents the automation strategies and decision-making processes for the curation pipeline of large infrared survey datasets, enhancing efficiency and consistency.

Findings

Automated pipeline reduces manual curation effort.

Supports diverse survey types with varying requirements.

Facilitates handling of billions of detections in large datasets.

Abstract

The two fastest near infrared survey telescopes are UKIRT-WFCAM and VISTA. The data from both these instruments are being archived by Wide Field Astronomy Unit (WFAU) at the IfA, Edinburgh, using the same curation pipeline, with some instrument specific processing. The final catalogues from these surveys will contain many tens of billions of detections. Data are taken for a range of large surveys and smaller PI programmes. The surveys vary from shallow hemisphere surveys to ultra deep single pointings with hundreds of individual epochs, each with a wide range of scientific goals, leading to a wide range of products and database tables being created. Processing of the main surveys must allow for the inclusion of specific high-level requirements from the survey teams, but automation reduces the amount of work by archive operators allowing a higher curation efficiency. The decision making processes which drive the curation pipeline are a crucial element for efficient archiving. This paper describes the main issues involved in automating the pipeline.