The swiss army knife of job submission tools: grid-control
F. Stober, M. Fischer, P. Schleper, H. Stadie, C. Garbers, J. Lange,, N. Kovalchuk

TL;DR
Grid-control is a versatile, open-source job submission tool designed for high energy physics, supporting complex workflows, various batch systems, and dynamic dataset management, enhancing efficiency and flexibility in large-scale analyses.
Contribution
It introduces a highly portable, plugin-based system that simplifies complex workflow specification and dataset management in high energy physics computing environments.
Findings
Supports up to 100,000 jobs in analyses
Flexible plugin system for workflow customization
Efficient dataset processing and job management
Abstract
Grid-control is a lightweight and highly portable open source submission tool that supports virtually all workflows in high energy physics (HEP). Since 2007 it has been used by a sizeable number of HEP analyses to process tasks that sometimes consist of up 100k jobs. grid-control is built around a powerful plugin and configuration system, that allows users to easily specify all aspects of the desired workflow. Job submission to a wide range of local or remote batch systems or grid middleware is supported. Tasks can be conveniently specified through the parameter space that will be processed, which can consist of any number of variables and data sources with complex dependencies on each other. Dataset information is processed through a configurable pipeline of dataset filters, partition plugins and partition filters. The partition plugins can take the number of files, size of the work…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
