Fast Hyperparameter Optimization of Deep Neural Networks via Ensembling Multiple Surrogates

TL;DR

This paper introduces HOIST, a hyperparameter optimization method that leverages both complete and intermediate evaluation data using ensemble surrogates to improve efficiency in tuning deep neural networks.

Contribution

HOIST is a novel approach that combines multiple surrogates trained on mixed evaluation data, including early stopping information, to accelerate DNN hyperparameter tuning.

Findings

HOIST outperforms state-of-the-art methods across various DNN architectures.

Utilizes both complete and intermediate evaluation data for better surrogate modeling.

Ensemble surrogates improve the accuracy and efficiency of hyperparameter optimization.

Abstract

The performance of deep neural networks crucially depends on good hyperparameter configurations. Bayesian optimization is a powerful framework for optimizing the hyperparameters of DNNs. These methods need sufficient evaluation data to approximate and minimize the validation error function of hyperparameters. However, the expensive evaluation cost of DNNs leads to very few evaluation data within a limited time, which greatly reduces the efficiency of Bayesian optimization. Besides, the previous researches focus on using the complete evaluation data to conduct Bayesian optimization, and ignore the intermediate evaluation data generated by early stopping methods. To alleviate the insufficient evaluation data problem, we propose a fast hyperparameter optimization method, HOIST, that utilizes both the complete and intermediate evaluation data to accelerate the hyperparameter optimization of DNNs. Specifically, we train multiple basic surrogates to gather information from the mixed evaluation data, and then combine all basic surrogates using weighted bagging to provide an accurate ensemble surrogate. Our empirical studies show that HOIST outperforms the state-of-the-art approaches on a wide range of DNNs, including feed forward neural networks, convolutional neural networks, recurrent neural networks, and variational autoencoder.