Themis: Fair and Efficient GPU Cluster Scheduling

TL;DR

Themis is a novel GPU cluster scheduling framework that ensures fair and efficient resource allocation for ML workloads by using a finish-time fairness policy and an auction-based two-level scheduling architecture.

Contribution

Themis introduces a new finish-time fairness policy and an auction-based scheduling architecture tailored for ML workloads' unique attributes.

Findings

Improves fairness by over 2.25 times compared to existing schedulers.

Achieves 5% to 250% higher cluster efficiency.

Effectively handles long-running, placement-sensitive ML tasks.

Abstract

Modern distributed machine learning (ML) training workloads benefit significantly from leveraging GPUs. However, significant contention ensues when multiple such workloads are run atop a shared cluster of GPUs. A key question is how to fairly apportion GPUs across workloads. We find that established cluster scheduling disciplines are a poor fit because of ML workloads' unique attributes: ML jobs have long-running tasks that need to be gang-scheduled, and their performance is sensitive to tasks' relative placement. We propose Themis, a new scheduling framework for ML training workloads. It's GPU allocation policy enforces that ML workloads complete in a finish-time fair manner, a new notion we introduce. To capture placement sensitivity and ensure efficiency, Themis uses a two-level scheduling architecture where ML workloads bid on available resources that are offered in an auction run by a central arbiter. Our auction design allocates GPUs to winning bids by trading off efficiency for fairness in the short term but ensuring finish-time fairness in the long term. Our evaluation on a production trace shows that Themis can improve fairness by more than 2.25X and is ~5% to 250% more cluster efficient in comparison to state-of-the-art schedulers.