Towards General Distributed Resource Selection

TL;DR

This paper presents a general resource selection model for distributed computing that estimates task costs across heterogeneous resources, improving workload efficiency significantly.

Contribution

It introduces a resource-agnostic cost model integrated with Condor for effective resource selection in heterogeneous environments.

Findings

Cost estimation error: 157-171% on XSEDE, 18-31% on OSG

Resource selection reduces workload time-to-completion by up to ~85%

Model enables efficient scheduling of large-scale GROMACS simulations

Abstract

The advantages of distributing workloads and utilizing multiple distributed resources are now well established. The type and degree of heterogeneity of distributed resources is increasing, and thus determining how to distribute the workloads becomes increasingly difficult, in particular with respect to the selection of suitable resources. We formulate and investigate the resource selection problem in a way that it is agnostic of specific task and resource properties, and which is generalizable to range of metrics. Specifically, we developed a model to describe the requirements of tasks and to estimate the cost of running that task on an arbitrary resource using baseline measurements from a reference machine. We integrated our cost model with the Condor matchmaking algorithm to enable resource selection. Experimental validation of our model shows that it provides execution time estimates with 157-171% error on XSEDE resources and 18-31% on OSG resources. We use the task execution cost model to select resources for a bag-of-tasks of up to 1024 GROMACS MD simulations across the target resources. Experiments show that using the model's estimates reduces the workload's time-to-completion up to ~85% when compared to the random distribution of workload across the same resources.