The Dark Energy Survey Data Management System
Joseph J. Mohr (1), Wayne Barkhouse (2), Cristina Beldica (1),, Emmanuel Bertin (3), Y. Dora Cai (1), Luiz da Costa (4), J. Anthony Darnell, (1), Gregory E. Daues (1), Michael Jarvis (5), Michelle Gower (1), Huan Lin, (6), leandro Martelli (4), Eric Neilsen (6)

TL;DR
The Dark Energy Survey Data Management System is a comprehensive, high-performance platform designed to process, archive, and distribute large-scale astronomical survey data efficiently, supporting both research and public access.
Contribution
This paper introduces the DESDM system, integrating HPC environments for efficient processing and management of large astronomical datasets, with successful testing on simulated and real survey data.
Findings
Processed 10 simulated survey nights with 3TB data successfully.
Calibrated over 50 nights of real survey data with high data quality.
Demonstrated system's capability for large-scale, high-quality data processing.
Abstract
The Dark Energy Survey collaboration will study cosmic acceleration with a 5000 deg2 griZY survey in the southern sky over 525 nights from 2011-2016. The DES data management (DESDM) system will be used to process and archive these data and the resulting science ready data products. The DESDM system consists of an integrated archive, a processing framework, an ensemble of astronomy codes and a data access framework. We are developing the DESDM system for operation in the high performance computing (HPC) environments at NCSA and Fermilab. Operating the DESDM system in an HPC environment offers both speed and flexibility. We will employ it for our regular nightly processing needs, and for more compute-intensive tasks such as large scale image coaddition campaigns, extraction of weak lensing shear from the full survey dataset, and massive seasonal reprocessing of the DES data. Data products…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
