The Dark Energy Survey Data Management System: The Processing Framework
Michelle Gower, Joseph J. Mohr, Darren Adams, Y. Dora Cai, Gregory E., Daues, Tony Darnell, Chow-Choong Ngeow, Shantanu Desai, Cristina Beldica,, Mike Freemon, Huan Lin, Eric H. Neilsen, Douglas Tucker, Emmanuel Bertin,, Luiz A. Nicolaci da Costa, Leandro Martelli
TL;DR
This paper presents a new adaptable processing framework for the Dark Energy Survey Data Management system, enabling automated, high-performance data processing and archiving over the survey's five-year operation.
Contribution
It introduces a novel processing framework designed for high automation and performance, specifically tailored for large-scale astronomical data from the Dark Energy Survey.
Findings
Successfully processed 45 nights of simulated data
Extensively used in Data Challenge 4 for large-scale data processing
Demonstrated high automation and performance in data handling
Abstract
The Dark Energy Survey Data Management (DESDM) system will process and archive the data from the Dark Energy Survey (DES) over the five year period of operation. This paper focuses on a new adaptable processing framework developed to perform highly automated, high performance data parallel processing. The new processing framework has been used to process 45 nights of simulated DECam supernova imaging data, and was extensively used in the DES Data Challenge 4, where it was used to process thousands of square degrees of simulated DES data.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAstronomy and Astrophysical Research · Gamma-ray bursts and supernovae · Astronomical Observations and Instrumentation