An intelligent Data Delivery Service for and beyond the ATLAS experiment
Wen Guan, Tadashi Maeno, Brian Paul Bockelman, Torre Wenaus, Fahui, Lin, Siarhei Padolski, Rui Zhang, Aleksandr Alekseev
TL;DR
The paper introduces the intelligent Data Delivery Service (iDDS), a workflow-oriented system designed to efficiently manage data pre-processing, delivery, and processing for large-scale experiments like ATLAS, accommodating future growth.
Contribution
It presents the design, architecture, and use cases of iDDS, a novel, experiment-agnostic data management system tailored for high-energy physics workflows.
Findings
iDDS effectively manages large data workflows in ATLAS.
The system is adaptable to various experiments and use cases.
Future plans aim to enhance scalability and functionality.
Abstract
The intelligent Data Delivery Service (iDDS) has been developed to cope with the huge increase of computing and storage resource usage in the coming LHC data taking. iDDS has been designed to intelligently orchestrate workflow and data management systems, decoupling data pre-processing, delivery, and main processing in various workflows. It is an experiment-agnostic service around a workflow-oriented structure to work with existing and emerging use cases in ATLAS and other experiments. Here we will present the motivation for iDDS, its design schema and architecture, use cases and current status, and plans for the future.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.