GlideinBenchmark: collecting resource information to optimize provisioning

TL;DR

GlideinBenchmark is a web application that streamlines resource benchmarking and enables automated, optimized resource selection for workload management in cloud and grid environments.

Contribution

It introduces a new benchmarking tool integrated with GlideinWMS to automate resource evaluation and improve provisioning decisions.

Findings

Automates resource benchmarking process.

Enables data-driven resource selection.

Improves job completion times and cost efficiency.

Abstract

Choosing the right resource can speed up job completion, better utilize the available hardware, and visibly reduce costs, especially when renting computers in the cloud. This was demonstrated in earlier studies on HEPCloud. However, the benchmarking of the resources proved to be a laborious and time-consuming process. This paper presents GlideinBenchmark, a new Web application leveraging the pilot infrastructure of GlideinWMS to benchmark resources, and it shows how to use the data collected and published by GlideinBenchmark to automate the optimal selection of resources. An experiment can select the benchmark or the set of benchmarks that most closely evaluate the performance of its workflows. GlideinBenchmark, with the help of the GlideinWMS Factory, controls the benchmark execution. Finally, a scheduler like HEPCloud's Decision Engine can use the results to optimize resource provisioning.