Bidding policies for market-based HPC workflow scheduling

TL;DR

This paper evaluates market-based scheduling policies for HPC clusters with dependent tasks, finding that the Projected Value Remaining policy maximizes platform value and minimizes task starvation under overload conditions.

Contribution

It introduces and compares bidding policies for market-based HPC scheduling, highlighting the effectiveness of the Projected Value Remaining policy in overload scenarios.

Findings

Projected Value Remaining policy yields highest platform value.

It minimizes task starvation during overload.

Some alternative policies achieve higher value but cause more starvation.

Abstract

This paper considers the scheduling of jobs on distributed, heterogeneous High Performance Computing (HPC) clusters. Market-based approaches are known to be efficient for allocating limited resources to those that are most prepared to pay. This context is applicable to an HPC or cloud computing scenario where the platform is overloaded. In this paper, jobs are composed of dependent tasks. Each job has a non-increasing time-value curve associated with it. Jobs are submitted to and scheduled by a market-clearing centralised auctioneer. This paper compares the performance of several policies for generating task bids. The aim investigated here is to maximise the value for the platform provider while minimising the number of jobs that do not complete (or starve). It is found that the Projected Value Remaining bidding policy gives the highest level of value under a typical overload situation, and gives the lowest number of starved tasks across the space of utilisation examined. It does this by attempting to capture the urgency of tasks in the queue. At high levels of overload, some alternative algorithms produce slightly higher value, but at the cost of a hugely higher number of starved workflows.