Hyper-Tune: Towards Efficient Hyper-parameter Tuning at Scale

TL;DR

Hyper-Tune is a scalable distributed hyper-parameter tuning framework that incorporates resource management, asynchronous scheduling, and multi-fidelity optimization to significantly improve efficiency and performance across various machine learning models.

Contribution

The paper introduces Hyper-Tune, a novel hyper-parameter tuning system with system-level optimizations for improved scalability and efficiency in real-world and benchmark scenarios.

Findings

Hyper-Tune outperforms existing systems on diverse models.

Achieves up to 11.2x speedup over BOHB.

Effective in large-scale production environments.

Abstract

The ever-growing demand and complexity of machine learning are putting pressure on hyper-parameter tuning systems: while the evaluation cost of models continues to increase, the scalability of state-of-the-arts starts to become a crucial bottleneck. In this paper, inspired by our experience when deploying hyper-parameter tuning in a real-world application in production and the limitations of existing systems, we propose Hyper-Tune, an efficient and robust distributed hyper-parameter tuning framework. Compared with existing systems, Hyper-Tune highlights multiple system optimizations, including (1) automatic resource allocation, (2) asynchronous scheduling, and (3) multi-fidelity optimizer. We conduct extensive evaluations on benchmark datasets and a large-scale real-world dataset in production. Empirically, with the aid of these optimizations, Hyper-Tune outperforms competitive hyper-parameter tuning systems on a wide range of scenarios, including XGBoost, CNN, RNN, and some architectural hyper-parameters for neural networks. Compared with the state-of-the-art BOHB and A-BOHB, Hyper-Tune achieves up to 11.2x and 5.1x speedups, respectively.