HyperSched: Dynamic Resource Reallocation for Model Development on a Deadline

TL;DR

HyperSched is a dynamic resource scheduler that reallocates resources during hyperparameter searches to maximize model accuracy before a deadline, outperforming traditional methods by leveraging trial disposability, ranking, and constraints.

Contribution

It introduces HyperSched, a novel scheduler that dynamically reallocates resources based on trial performance, optimizing hyperparameter search under time constraints.

Findings

HyperSched outperforms standard hyperparameter search algorithms.

It effectively identifies and allocates resources to promising trials.

The scheduler leverages properties like trial disposability and ranking for efficiency.

Abstract

Prior research in resource scheduling for machine learning training workloads has largely focused on minimizing job completion times. Commonly, these model training workloads collectively search over a large number of parameter values that control the learning process in a hyperparameter search. It is preferable to identify and maximally provision the best-performing hyperparameter configuration (trial) to achieve the highest accuracy result as soon as possible. To optimally trade-off evaluating multiple configurations and training the most promising ones by a fixed deadline, we design and build HyperSched -- a dynamic application-level resource scheduler to track, identify, and preferentially allocate resources to the best performing trials to maximize accuracy by the deadline. HyperSched leverages three properties of a hyperparameter search workload over-looked in prior work - trial disposability, progressively identifiable rankings among different configurations, and space-time constraints - to outperform standard hyperparameter search algorithms across a variety of benchmarks.