DEEP-BO for Hyperparameter Optimization of Deep Networks

TL;DR

DEEP-BO is a specialized Bayesian optimization algorithm designed to efficiently tune hyperparameters of deep neural networks, outperforming or matching existing methods across multiple benchmarks.

Contribution

The paper introduces DEEP-BO, a novel hyperparameter optimization method tailored for deep networks, incorporating diversification, early stopping, and parallelization strategies.

Findings

DEEP-BO outperforms several existing hyperparameter optimization methods.

DEEP-BO achieves comparable results with state-of-the-art solutions.

The approach is validated on six deep learning benchmarks.

Abstract

The performance of deep neural networks (DNN) is very sensitive to the particular choice of hyper-parameters. To make it worse, the shape of the learning curve can be significantly affected when a technique like batchnorm is used. As a result, hyperparameter optimization of deep networks can be much more challenging than traditional machine learning models. In this work, we start from well known Bayesian Optimization solutions and provide enhancement strategies specifically designed for hyperparameter optimization of deep networks. The resulting algorithm is named as DEEP-BO (Diversified, Early-termination-Enabled, and Parallel Bayesian Optimization). When evaluated over six DNN benchmarks, DEEP-BO easily outperforms or shows comparable performance with some of the well-known solutions including GP-Hedge, Hyperband, BOHB, Median Stopping Rule, and Learning Curve Extrapolation. The code used is made publicly available at https://github.com/snu-adsl/DEEP-BO.

Tables

Table 1. Table 1 : Benchmark problem settings.

HPO Benchmark	Number of hyperparameters			Learning rate scale	Max epoch	Number of configurations
	discrete	continuous	categorical
MNIST-LeNet1	6	3	-	log	15	20,000
MNIST-LeNet2	3	3	3	log	15	20,000
PTB-LSTM	3	5	1	linear	15	20,000
CIFAR10-VGG	6	2	2	log	50	20,000
CIFAR10-ResNet	2	4	1	log	100	7,000
CIFAR100-VGG	6	2	2	log	50	20000

Table 2. Table 2 : Summary of performance achieving top-10 accuracy. For success rate , the numbers in parenthesis of each benchmark indicate the checkpoints when the number of processors is 1 and 6, respectively. We only show the competitors’ performance due to the page limit. See Appendix D for full benchmark results.

Measure	Benchmark	Single processor ($𝑴=\mathrm{𝟏}$)				Six processors ($𝑴=\mathrm{𝟔}$)
		GP-Hedge	GP-EI-MSR	BOHB	DEEP-BO	GP-EI	GP-EI-MSR	DEEP-BO
Success rate	MNIST-LeNet1 (12h / 2h)	32%	32%	12%	81%	15%	15%	80%
	MNIST-LeNet2 (12h / 2h)	39%	48%	1%	82%	39%	41%	95%
	PTB-LSTM (12h / 2h)	85%	89%	29%	100%	45%	60%	96%
	CIFAR10-VGG (6h / 1h)	53%	48%	0%	80%	32%	50%	75%
	CIFAR10-ResNet (90h / 15h)	91%	94%	4%	100%	61%	81%	99%
	CIFAR100-VGG (6h / 1h)	92%	85%	4%	100%	30%	43%	99%
	Mean	66%	66%	8%	91%	37%	48%	91%
Expected time (hour)	MNIST-LeNet1	$17.0\pm 8.2$	$16.3\pm 8.6$	$20.7\pm 6.4$	$8.2\pm 8.4$	$10.4\pm 8.5$	$4.0\pm 1.5$	$1.4\pm 1.0$
	MNIST-LeNet2	$29.3\pm 31.2$	$24.2\pm 30.4$	$35.8\pm 12.1$	$7.6\pm 10.4$	$4.6\pm 5.4$	$2.4\pm 1.4$	$0.8\pm 1.0$
	PTB-LSTM	$8.3\pm 3.6$	$7.9\pm 2.8$	$28.3\pm 22.7$	$4.7\pm 2.2$	$2.1\pm 0.7$	$1.8\pm 0.8$	$1.0\pm 0.4$
	CIFAR10-VGG	$6.6\pm 4.0$	$6.8\pm 4.3$	$29.4\pm 1.8$	$3.4\pm 2.5$	$1.5\pm 0.9$	$1.3\pm 0.8$	$0.8\pm 0.4$
	CIFAR10-ResNet	$50.6\pm 29.6$	$43.8\pm 25.4$	$129.8\pm 17.8$	$38.4\pm 17.3$	$13.0\pm 5.5$	$11.4\pm 4.1$	$7.7\pm 2.5$
	CIFAR100-VGG	$4.1\pm 2.1$	$3.5\pm 1.5$	$24.8\pm 9.9$	$1.9\pm 0.9$	$1.2\pm 0.5$	$1.1\pm 0.4$	$0.5\pm 0.2$
	Mean	$25.7$	$17.2$	$44.8$	$10.7$	$5.5$	3.7%	$2.0$

Table 3. Table 3 : Ablation test results with DEEP-BO using 6 processors. Bold refers to the best performance and Italic refers to comparable results.

Hybrid Transformation	Early Termination	In- progress	Expected time (hour)
			MNIST- LeNet1	MNIST- LeNet2	PTB- LSTM	CIFAR10- VGG	CIFAR10- ResNet	CIFAR100- VGG
on	on	on	$1.4\pm 1.0$	$0.8\pm 1.0$	$1.0\pm 0.4$	$0.8\pm 0.4$	$7.7\pm 2.5$	$0.5\pm 0.2$
on	on	off	$1.6\pm 1.0$	$0.7\pm 0.5$	$1.1\pm 0.5$	$0.9\pm 0.6$	$10.5\pm 3.8$	$0.7\pm 0.3$
on	off	on	$1.3\pm 0.7$	$1.0\pm 1.5$	$1.2\pm 0.5$	$0.9\pm 0.6$	$8.6\pm 2.9$	$0.8\pm 0.3$
on	off	off	$1.8\pm 1.0$	$1.4\pm 1.5$	$1.4\pm 0.6$	$1.4\pm 0.8$	$11.6\pm 4.1$	$1.1\pm 0.3$
off	on	on	$2.2\pm 1.7$	$1.0\pm 0.8$	$1.5\pm 0.7$	$0.7\pm 0.5$	$8.9\pm 3.3$	$0.5\pm 0.2$
off	on	off	$2.7\pm 2.0$	$1.2\pm 1.0$	$1.7\pm 0.8$	$1.0\pm 0.7$	$10.6\pm 4.4$	$0.8\pm 0.3$
off	off	on	$2.4\pm 1.8$	$1.5\pm 1.1$	$1.8\pm 0.7$	$1.1\pm 0.7$	$10.5\pm 4.4$	$0.8\pm 0.3$
off	off	off	$2.4\pm 1.6$	$1.8\pm 1.3$	$2.1\pm 0.9$	$1.2\pm 0.8$	$12.7\pm 4.9$	$1.1\pm 0.4$

Table 4. Table B1. Each cell shows μ ± σ plus-or-minus 𝜇 𝜎 \mu\pm\sigma of the respective median survivor rank regret values for 3 pre-evaluated surrogate configurations(MNIST-LeNet2, PTB-LSTM, and CIFAR10-VGG benchmark) for 50 trials with our ETR on the cell’s setting(row: s 2 subscript 𝑠 2 s_{2} , column: e 2 subscript 𝑒 2 e_{2} ). As DEEP-BO uses β = 0.1 𝛽 0.1 \beta=0.1 , this table sets the threshold at the 90th percentile value for comparison.

BO	GP-EI			RF-EI
	50%	70%	90%	50%	70%	90%
0%	$13.2\pm 5.1$%	$13.2\pm 5.2$%	$12.7\pm 4.2$%	$4.8\pm 4.8$%	$4.2\pm 4.0$%	$3.7\pm 2.9$%
10%	$11.3\pm 3.2$%	$11.2\pm 3.4$%	$10.6\pm 2.5$%	$2.9\pm 2.6$%	$3.0\pm 2.3$%	$2.9\pm 2.2$%
20%	$9.4\pm 1.5$%	$9.6\pm 1.2$%	$9.2\pm 1.1$%	$1.5\pm 0.9$%	$1.9\pm 1.2$%	$1.7\pm 1.0$%
30%	$8.0\pm 1.1$%	$8.8\pm 0.6$%	$8.6\pm 0.8$%	$1.1\pm 0.6$%	$1.2\pm 0.7$%	$1.3\pm 0.7$%
40%	$7.1\pm 0.3$%	$7.4\pm 0.5$%	$7.9\pm 0.5$%	$0.5\pm 0.4$%	$0.8\pm 0.3$%	$0.9\pm 0.4$%

Table 5. Table C1: The hyperparameter spaces of LeNet on MNIST dataset.

Architecture	Name	Type	Range
			LeNet1	LeNet2
LeNet-5	\pbox20cmnumber of first convolution
kernels	discrete	[1, 350]	[1, 350]
	\pbox10cmfirst pooling size	discrete	[2, 3]	-
	\pbox10cmnumber of second convolution
kernels	discrete	[1, 350]	[1, 350]
	\pbox10cmsecond pooling size	discrete	[2, 3]	-
	\pbox10cmnumber of neurons in fully
connected layer	discrete	[1, 1024]	[1, 1024]
	\pbox10cmsquare length of the convolution
kernel	discrete	[2, 10]	-
	learning rate	continuous	\pbox10cm[0.0001, 0.4]
(log scale)	\pbox10cm[0.0001, 0.4]
(log scale)
	L2 regularization factor	continuous	[0.0, 1.0]	[0.0, 1.0]
	drop out rate	continuous	[0.0, 0.9]	[0.0, 1.0]
	activation function type	categorical	-	\pbox10cmReLU, tanh,sigmoid,
eLU, leaky ReLU
	optimizer type	categorical	-	\pbox10cmAdaDelta, AdaGrad, Adam,
GD, Momentum, RMSProp
	batch normalization	categorical	-	\pbox10cmEnable, Disable

Table 6. Table C2: The hyperparameter spaces of VGGNet and ResNet on CIFAR10.

Architecture	Name	Type	Range
VGG	\pbox10cmnumber of first convolution kernels	discrete	[8, 32]
	\pbox10cmnumber of second convolution kernels	discrete	[32, 64]
	\pbox10cmnumber of third convolution kernels	discrete	[64, 128]
	\pbox10cmnumber of forth convolution kernels	discrete	[64, 128]
	\pbox10cmnumber of neurons in fully connected layer	discrete	[10, 1000]
	\pbox10cmsquare length of convolution kernel	discrete	[2, 3]
	learning rate	continuous	\pbox10cm[0.0001, 0.4]
(log scale)
	L2 regularization factor	continuous	[0.0, 1.0]
	activation function type	categorical	\pbox10cmReLU, tanh, eLU
	regularization method	categorical	\pbox10cmNone, Dropout, BatchNorm
ResNet	\pbox10cmnumber of layers	discrete	38, 44, 50, 56, 62, 70, 78
	\pbox10cmtraining data batch size	discrete	45, 90, 180, 360, 450
	\pbox10cmdata augmentation	categorical	True, False
	learning rate	continuous	\pbox10cm[0.0001, 0.1]
(log scale)
	\pbox10cmmomentum for MomentumOptimizer	continuous	[0.1, 0.9]
	\pbox10cmweight decay	continuous	\pbox10cm[0.00001, 0.001]
(log scale)
	\pbox10cmbatch normalization decay	continuous	\pbox10cm[0.9, 0.999]

Table 7. Table C3: The hyperparameter space of LSTM on PTB dataset.

Architecture	Name	Type	Range
LSTM	\pbox10cmnumber of neurons in hidden layer	discrete	[10, 200]
	\pbox10cmnumber of hidden layers	discrete	[1, 2]
	\pbox10cmnumber of training steps	discrete	[10, 20]
	\pbox10cminitial value of uniform random scale	continuous	[0.01, 0.1]
	drop out rate	continuous	[0.0, 0.9]
	learning rate	continuous	\pbox10cm[0.1, 1.0]
	learning rate decay	continuous	[0.5, 1.0]
	\pbox10cmgradient clipping by global normalization	continuous	[5.0, 10.0]
	RNN training module	categorical	\pbox10cmBasic, Block, cuDNN

Table 8. Table C4: The hyperparameter space of VGGNet on CIFAR100 dataset.

Architecture	Name	Type	Range
VGG	\pbox10cmnumber of first convolution kernels	discrete	[8, 32]
	\pbox10cmnumber of second convolution kernels	discrete	[32, 64]
	\pbox10cmnumber of third convolution kernels	discrete	[64, 128]
	\pbox10cmnumber of forth convolution kernels	discrete	[64, 128]
	\pbox10cmnumber of neurons in fully connected layer	discrete	[10, 1000]
	\pbox10cmsquare length of convolution kernel	discrete	[2, 3]
	learning rate	continuous	\pbox10cm[0.0001, 0.4]
(log scale)
	L2 regularization factor	continuous	[0.0, 1.0]
	activation function type	categorical	\pbox10cmReLU, tanh,sigmoid, eLU
	regularization method	categorical	\pbox10cmNone, Dropout, BatchNorm

Table 9. Table C5: Statistics of pre-evaluated configurations.

Dataset	MNIST		PTB	CIFAR10		CIFAR100
Architecture	LeNet1	LeNet2	LSTM	VGG	ResNet	VGG
Total evaluation time (day)	88.1	55.5	87.3	75.2	385	82.1
Mean evaluation time (min)	6.3	4	6.3	5.4	79	5.9
Global optimum (top accuracy)	0.9939	0.9941	0.9008	0.8052	0.9368	0.5067
Target performance (top-10 accuracy)	0.9933	0.9922	0.8993	0.7836	0.9333	0.4596

Table 10. Table D1. Summary of single processor performance achieving top-10 accuracy.

Measure	Benckmark	GP-EI	GP-PI	GP-UCB	RF-EI	RF-PI	RF-UCB
Success rate	MNIST-LeNet1 (12h)	34%	34%	29%	48%	61%	41%
	MNIST-LeNet2 (12h)	49%	29%	48%	63%	57%	65%
	PTB-LSTM (12h)	89%	83%	86%	68%	92%	89%
	CIFAR10-VGG (6h)	50%	43%	46%	77%	77%	75%
	CIFAR10-ResNet (90h)	91%	86%	92%	69%	89%	86%
	CIFAR100-VGG (6h)	87%	81%	79%	96%	98%	97%
	Mean	67%	59%	63%	70%	79%	76%
Expected time (hour)	MNIST-LeNet1	$53.1\pm 74.4$	$56.9\pm 81.5$	$67.5\pm 84.4$	$43.2\pm 59.7$	$27.8\pm 65.2$	$36\pm 51.7$
	MNIST-LeNet2	$29.6\pm 46.7$	$77.9\pm 76.6$	$25.3\pm 27.5$	$16.5\pm 23.8$	$45.4\pm 78.5$	$20.4\pm 34.3$
	PTB-LSTM	$8.2\pm 3.4$	$7.9\pm 4.6$	$8.6\pm 3.4$	$25.8\pm 61.4$	$6.4\pm 3.4$	$7.4\pm 3.5$
	CIFAR10-VGG	$7.4\pm 4.3$	$7.2\pm 3.5$	$8.1\pm 4.7$	$4.7\pm 3.2$	$4.4\pm 3.8$	$4.4\pm 3.5$
	CIFAR10-ResNet	$44.1\pm 28.9$	$57.2\pm 32.4$	$47.5\pm 24.6$	$78.5\pm 59.6$	$53.5\pm 35.2$	$55.7\pm 30.4$
	CIFAR100-VGG	$4.1\pm 1.6$	$4.1\pm 2.1$	$6.7\pm 11.7$	$3.5\pm 6.3$	$3.2\pm 7.2$	$3.1\pm 5.5$
	Mean	$24.4$	$35.2$	$27.3$	$28.7$	$23.4$	$21.2$

Table 11. Table D3. Summary of the 6 processors performance achieving top-10 accuracy.

Measure	Benckmark	GP-EI	GP-PI	GP-UCB	RF-EI	RF-PI	RF-UCB	GP-EI-MSR	DEEP-BO
Success rate	MNIST-LeNet1 (2h)	15%	9%	13%	32%	58%	30%	15%	80%
	MNIST-LeNet2 (2h)	39%	17%	38%	44%	56%	46%	41%	95%
	PTB-LSTM (2h)	45%	68%	52%	24%	63%	35%	60%	95%
	CIFAR10-VGG (1h)	32%	43%	39%	40%	48%	32%	50%	75%
	CIFAR10-ResNet (15h)	61%	59%	70%	37%	58%	55%	81%	99%
	CIFAR100-VGG (1h)	30%	25%	37%	51%	59%	47%	43%	99%
	Mean	37%	37%	42%	38%	57%	41%	48%	91%
Expected time (hour)	MNIST-LeNet1	$10.4\pm 8.5$	$11.9\pm 8.9$	$8.9\pm 7.8$	$4.7\pm 5.2$	$2.4\pm 2.0$	$4.6\pm 4.3$	$4.0\pm 1.5$	$1.4\pm 1.0$
	MNIST-LeNet2	$4.6\pm 5.4$	$8.4\pm 7.3$	$3.8\pm 3.8$	$3.0\pm 2.8$	$3.3\pm 4.5$	$2.4\pm 1.9$	$2.4\pm 1.4$	$0.8\pm 1.0$
	PTB-LSTM	$2.1\pm 0.7$	$1.7\pm 0.8$	$2.0\pm 0.7$	$2.5\pm 0.7$	$1.8\pm 0.9$	$2.3\pm 0.8$	$1.8\pm 0.8$	$1.0\pm 0.4$
	CIFAR10-VGG	$1.5\pm 0.9$	$1.5\pm 1.1$	$1.5\pm 1.0$	$1.5\pm 1.0$	$1.4\pm 1.0$	$1.8\pm 1.1$	$1.3\pm 0.8$	$0.8\pm 0.4$
	CIFAR10-ResNet	$13.0\pm 5.5$	$14.1\pm 6.6$	$12.3\pm 5.7$	$18.1\pm 7.3$	$14.4\pm 6.6$	$14.6\pm 6.6$	$11.4\pm 4.1$	$7.7\pm 2.5$
	CIFAR100-VGG	$1.2\pm 0.5$	$1.4\pm 0.5$	$1.2\pm 0.5$	$1.1\pm 0.4$	$1.1\pm 1.0$	$1.0\pm 0.4$	$1.1\pm 0.4$	$0.5\pm 0.2$
	Mean	5.5	6.5	5.0	5.2	4.1	4.4	3.7	2.0

Equations

$$g(y,\alpha )=\{\begin{array}{ll}y& \text{if }y<1-\alpha \\ 1-\log (1-y)+(\log (\alpha )-\alpha ),& \text{otherwise}\end{array}$$

$$F_X(x)$$

$${\bar{f}}_{i:j}(\mathbf{x})$$

$$l(E,\beta )$$

$$S$$

$$h(j,\mathcal{A},\mathcal{H},\beta )=\{\begin{array}{ll}{F}_X^{-1}(\beta )& \text{if }j\le j_1\\ F_X^{-1}(1-\beta )& \text{otherwise}\end{array}\text{for,}$$

$$X=\{\begin{array}{ll}\{x\mid x={\bar{f}}_{1:j_1}(\mathbf{x})\text{for}\mathbf{x}\in \mathcal{H}\}& \text{if }j\le j_1\\ \{x\mid x={\bar{f}}_{j_1:j_2}(\mathbf{x})\text{for}\mathbf{x}\in \mathcal{S}\}& \text{otherwise}\end{array}$$

$$q(\beta )=(0.5\beta +(1-\beta {)}^3+(1-\beta )\beta {)}^{-1}$$

$$L_{\Theta }:\mathbf{X}\times \mathbf{V}\to \mathbf{T}$$

Full text

1 Introduction

Hyperparameter optimization(HPO) aims to find the global optima $\mathbf{x}^{*}$ of an unknown black-box function $f$. That is, $\mathbf{x}^{*}=\operatorname*{arg\,max}_{\mathbf{x}\in\mathcal{X}}f(\mathbf{x})$, where $\mathcal{X}$ is a hyperparameter space that can be a subset of $\mathbb{R}^{d}$, contain categorical, or discrete variables. $f(\mathbf{x})$ can be evaluated at an arbitrary $\mathbf{x}\in\mathcal{X}$. Bayesian Optimization(BO) algorithms have been deeply studied and used for HPO in many machine learning(ML) and deep neural network(DNN) problems. Recently, DNN HPO has gained greater importance with increasingly complex tasks that ML techniques cannot fully support.

We have observed that it is difficult to choose an appropriate HPO algorithm because the search space depends on the data characteristics and the DNN structure. As shown in Figure 1, we first evaluated 3 different DNN tasks using various HPO algorithms: random search (Bergstra and Bengio, 2012), Gaussian Process(GP-EI) (Snoek et al., 2012), an adaptive GP algorithm GP-Hedge (Hoffman et al., 2011), two speed-up BOs using Median Stopping Rule(GP-EI-MSR) and Learning Curve Extrapolation(RF-EI-LCE) (Golovin et al., 2017; Domhan et al., 2015), bandit based approach Hyperband (Li et al., 2017) and BOHB (Falkner et al., 2018). No free lunch theorem proves that no algorithm can show superior performance in all cases (Wolpert et al., 1997). Nevertheless, DEEP-BO that this work introduces shows stable performance in most cases.

DEEP-BO (Diversified, Early-termination Enabled, Parallel BO) is an algorithm made more robust by strategically integrating 3 methods - BO diversification, safe early termination, and cost function transformation. We first detail how each method comprising DEEP-BO is developed to enhance its respective precedents and then proceed to observe how their combination outperforms existing strategies for DNN HPO. We discuss the individual strengths of the comprising methods and observe that their combined use in the form of DEEP-BO shows both greater performance and robustness than when used alone in enhancing traditional BO for DNN HPO tasks.

2 Related Works

2.1 Bayesian Optimization

Bayesian optimization is an iterative process that sequentially chooses the next configuration to evaluate based on previous observations. Three Sequential Model-Based Global Optimization (SMBO) algorithms, Spearmint (Snoek et al., 2012), SMAC (Hutter et al., 2011), and TPE (Bergstra et al., 2011), were introduced and are compared in practical problems (Eggensperger et al., 2013). Acquisition functions such as Probability of Improvement (PI), Expected Improvement (EI), and Upper Confidence Bound (UCB) are widely used with these three BO algorithms.

2.2 Diversification strategies

Diversification strategy is introduced to minimize the risk of loss by splitting different categories of methods in various problems. Diversification strategies have been successfully used in many fields such as communication (Zheng and Tse, 2003) and finance (Markowitz, 1952) because of their theoretical and practical advantages. Diversification strategies have also been applied to HPO of ML algorithms in various ways. GP-Hedge (Hoffman et al., 2011) uses multiple acquisition functions together in online multi-armed bandit settings to improve HPO performance. For situations where there is no clear reason to choose a particular acquisition function, using a portfolio of them can be beneficial.

2.3 Early stopping strategies

Speed up practices today attempt to reap the benefits of HPO in the form of resource allocation strategies by considering learning dynamics of DNN. One common prior is that learning curves typically increase and saturate. Li et al. (2017) further articulates this using envelope functions111It is defined as the maximum deviation of the intermediate losses from the terminal validation loss toward which the deviations are suggested to monotonically decrease with resources allocated. before applying it to the strategy. Even though asymptotic accuracy predictions vary from a heuristic rule (Golovin et al., 2017), probabilistic modeling (Domhan et al., 2015), to regression (Baker et al., 2017; Istrate et al., 2018), the termination criterion is set upon such an assumption. Some works develop on these precedents. Klein et al. (2016) uses Bayesian methods to  Domhan et al. (2015). Falkner et al. (2018) applies BO to Li et al. (2017) to capture the density of good configurations evaluated. Most of these works term their strategies as early stopping strategies.

2.4 Parallel BO

BO is a naturally sequential process, but parallelization can also be applied. Some studies address the following issues about parallel BO: scalability (Springenberg et al., 2016), batch configuration (Wu and Frazier, 2016; Nguyen et al., 2016), and exploration-exploitation tradeoffs (Desautels et al., 2014). Snoek et al. (2012) calculates Monte Carlo estimates of the acquisition function, which can be used in multiprocessor settings where the number of MCMC sampling can control the diversity of a GP model. This dynamic is briefly examined in this work. Many other strategies have been introduced to increase the diversity of parallel BO algorithms: combining UCB and pure exploration (Contal et al., 2013), using a penalized acquisition function (González et al., 2016), introducing a randomized strategy to divide and conquer (Wang et al., 2017) and modeling the diversity of a batch (Kathuria et al., 2016).

3 DEEP-BO: An Enhanced Bayesian Algorithm for DNN HPO

We introduce three methods intended to enhance BO performance and robustness. Appendix A details the full algorithm with more explanation.

3.1 Diversification of BO

While a typical BO algorithm uses a single modeling algorithm $\hat{f}$ for all iterations to approximate the true black-box function $f$, sequential diversification uses $N$ modeling algorithms ($\hat{f}_{1},\dotsc,\hat{f}_{N}$) in sequence. This approach rotates over the ordered $N$ algorithms in sequence. When $M$ parallel processors (workers) are available, the parallel diversification strategy can be used by utilizing $N$ modeling algorithms over $M$ processors. Assuming that the time required for evaluating $f(\mathbf{x}^{*})$ is much longer than that required for modeling $\hat{f}_{n}$ using history $\mathcal{H}$, which is often true for heavy DNN problems, we can adopt a simple strategy where we assign the modeling result $\mathbf{x}^{*}=\operatorname*{arg\,max}_{\mathbf{x}\in\mathcal{X}}\hat{f}_{n}(\mathbf{x})$ to whichever worker that becomes available. Here, $n$ would be cycled from 1 to $N$ in sequence such that diversification can be enforced.

3.2 Early termination rule

We introduce an early termination rule, which we name as the Compound Rule (CR), developed on the Median Stopping Rule (MSR) (Golovin et al., 2017) suggested to be robust. We define Early Termination Rule (ETR) as a rule that economizes on resources by prematurely terminating unpromising configurations 222Note that we avoided the term early stopping because historically it has been used to refer to a specific process that prevents overfitting from excessive training  (Prechelt, 1998).. MSR records the mean performance up to some epoch $j$ for each configuration tried and uses its median value to evaluate the pending configuration’s best performance until epoch $j$. Our ETR has 2 design parameters (max epoch $E$ and $\beta\in(0.0,0.5]$). $\beta=0.1$ is used here with less expected aggressiveness. CR considers curves to be one of three types - highly untrainable, highly desirable, and the rest - and aims to find the first two with checkpoint-1 and checkpoint-2 respectively. Section 5 and Appendix A and B contain further explanation this ETR. Below is a summary of the two checkpoints.

Checkpoint-1 discards highly untrainable configurations. It terminates the training of the current $k_{\text{th}}$ configuration $\mathbf{x}_{k}$ at epoch $j_{1}$ if its until-then best performance is strictly worse than the $\beta$ percentile value of the running averages $\bar{f}_{1:j_{1}}({\mathbf{x}})$ for all $\mathbf{x}\in{\mathcal{H}}$. The running average performance of $\mathbf{x}_{k}$ up to epoch $l$ is calculated as $\bar{f}_{1:l}({\mathbf{x}})=\frac{1}{l}\sum^{l}_{i=1}f_{i}({\mathbf{x}})$ where $f_{i}({\mathbf{x}})$ is performance of $\mathbf{x}$ at epoch $i$. Poor configurations consistently produce near random-guess performances and more likely arise when necessary priors for hyperparameters are unavailable which greatly limits the optimization potential (Gülçehre and Bengio, 2016). Checkpoint-1 exclusively aims to terminate them with a low ($\beta$ percentile; MSR uses 50th percentile) threshold.

Checkpoint-2 spares only highly desirable configurations that are likely to outperform the current best. $\mathbf{x}_{k}$ that survives checkpoint-1 is stopped at epoch $j_{2}$ if its until-then best performance is below the $(1-\beta)$ percentile value of the running averages $\bar{f}_{j_{1}:j_{2}}({\mathbf{x}})$ for $\mathbf{x}\in{S}$ where $S\subset{\mathcal{H}}$ for $\mathbf{x}$ that survived checkpoint-1. Envelope functions suggest more resources be needed to discern configuration quality since envelope overlaps decrease with resources more clearly. We observe this more evidently for highly desirable configurations. We thereby apply the intuition exclusively to discern these good configurations with $j_{2}$ proportional to threshold height. We also note that the observed range for checkpoint-2 starts from $j_{1}$, not the first epoch.

3.3 Parallel BO

As Snoek et al. (2012) points out, the main problem with parallel workers is deciding what $\mathbf{x}$ to evaluate next while a set of points are still being evaluated. The simplest strategy is to ignore others and include only completely evaluated results into $\mathcal{H}$. This can result in duplicate selection where the chosen $\mathbf{x}^{*}$ is already being evaluated by another worker. Snoek et al. (2012) adds randomness through MCMC sampling to reduce duplicate selection. In the case of parallelization, using $N$ different algorithms naturally lowers the chance of duplicates because the algorithms tend to make different decisions even for the same $\mathcal{H}$. When $M$ is much larger than $N$, however, duplicate selection becomes more likely even for diversification. When it occurs, the following can be considered.

• •

naïve: evaluate the duplicate anyway

• •

random: randomly select a new candidate

• •

next candidate: choose the next best candidate that has not been selected before

• •

in-progress: temporarily add the premature evaluation results to $\mathcal{H}$ and find $\mathbf{x}^{*}$

For DNN, in-progress is attractive because even when the premature result is entirely wrong, it is self-corrected and replaced in $\mathcal{H}$ when evaluation completes. Experiments conducted with $N=M=6$ shows that in-progress provides meaningful improvement while the others perform comparably.

Another important topic to consider is cost function transformation. For both improving modeling performance and robustness, we also introduce a hybrid transformation which uses a linear scale when the error rate is above $\alpha$ and log transformation when the error rate is below $\alpha$. DNN is often used for classification tasks with the performance as an error rate. When the desired error rate is close to 0, it becomes difficult to accurately model $\hat{f}(\mathbf{x})$ using the original linear scale. For such case, applying log transformation, known to be effective for modeling nonlinear and multimodal functions (Jones et al., 1998), can be advantageous. However, when the error rate is high, this log transformation can be harmful.

4 Empirical Results

In this study, we thoroughly evaluated DEEP-BO over 6 DNN HPO problems. As summarized in Table 1, we chose the most popular benchmark datasets in the DNN community: MNIST, PTB(Penn Treebank), CIFAR-10, and CIFAR-100. Three HPO tasks of CNN architectures (LeNet, VGG, ResNet) and one of RNN architecture (LSTM) are included. Since practical model implementation almost always limits the number of hyperparameters human experts can adjust, seven to ten hyperparameters are selected for optimization for each benchmark task. Learning rate, often the most critical, is included in all six problems. To repeat the experiments effectively over multiple algorithms, we first created lookup tables for each task by pre-evaluating surrogate configurations sampled by Sobol sequences Sobol’ (1967). The appendix C specifies more details of each benchmark.

We use two different metrics to evaluate HPO algorithms. The first is the success rate, the probability an HPO algorithm achieves a target goal within the given time budget $t$. Assuming the algorithm takes time $\tau$, a random variable, to achieve a target accuracy $c$, the algorithm’s success rate at time $t$ can be defined as successfully finding $\hat{\mathbf{x}}$ such that $f(\hat{\mathbf{x}})>c$ before time $t$, or more simply as $\mathbf{P}(\tau\leq t)$. The second is expected time to achieve a target accuracy expressed as $\mathbb{E}[\tau]$.

Figure  2 shows the success rate plots of DEEP-BO and other algorithms in 3 of 6 benchmark cases. Unlike other benchmark algorithms, DEEP-BO consistently shows top-level performance for all 3 benchmark cases shown. Appendix D of supplementary materials details plots for all benchmarks.

Even under the conditions of using a single processor, DEEP-BO is not only robust in all benchmarks, but also has excellent performance. In Figure  2(a)$\sim$2(c), we evaluated our method using $M$ = 1, $N$ = 6, $\alpha$ = 0.3, $\beta$ = 0.1 and compared it with an adaptive algorithm GP-Hedge, two speed up algorithms (MSR, LCE) and BOHB, a state-of-the-art algorithm by Falkner et al. (2018).

The performance of DEEP-BO under multiprocessing conditions is impressive. Figure 2(d)$\sim$2(f) plot success rates evaluated in 6 parallel processors($N$ = 6, $M$ = 6, $\alpha$ = 0.3, $\beta$ = 0.1). DEEP-BO is compared with four different settings of GP-EI algorithms regarding synchronization, MCMC sampling, and early termination. In GP-EI algorithms, MCMC sampling explained in Snoek et al. (2012) was adopted with either 10 or 1 samples used. The GP-EI-MCMC(10) was the baseline algorithm, and GP-EI-MCMC(1) used only one sample to increase randomness. ‘Synch,GP-EI-MCMC(10)’ was a special setting where all the six processors were synchronized for modeling such that there is no difficulty with handling incomplete evaluations. All of six processors started simultaneously and had to wait until the evaluations of all six completed such that the next top 6 candidates can be launched together. It performs poorly as expected, but is provided as a benchmark. Both GP-EI-MCMC(1) and GP-EI-MCMC(10) observably performed comparably and the increased randomness did not a prompt distinct improvement. MSR enabled GP-EI tends to increase the performance but, it is not significantly improved.

DEEP-BO notably enhances performance with a single processor. DEEP-BO with 6 processors can enhance more through parallelization. As shown in Figure 2, DEEP-BO outperforms in all benchmarks and seems robust than competitors. In terms of expected time, DEEP-BO performance using 1 processor is 10.7 hours, but that with 6 processors is only 2 hours (5.35 times faster). The state of the art algorithm, BOHB, did not perform well in these benchmarks even though we had tried to set its hyperparameters properly. We discuss why in the section 5.

The ablation study in Table 3 shows that, under all conditions, components applied to DEEP-BO do not always perform well but are generally stable. Table 3 shows (on, on, on) produces the most successful performances for 3 of 6 tasks and second best performances at worst. Using this with some components turned off can be a little harmful, but it is shown that all the best cases were where at least 2 of the 3 switches were on. Overall, (on, on, on) is a beneficial strategy for all benchmarks.

5 Discussion

A few of the core design principles of DEEP-BO are explained and discussed in this section.

5.1 Diversity gain and cooperation effect

For an HPO algorithm, we can define its failure rate as $p_{f}=1-\mathbf{P}(\tau\leq t)$ where $\tau$ is a random variable representing the algorithm’s time to achieve the target accuracy $c$. If the algorithm is run over $M$ parallel workers without sharing their histories $\{\mathcal{H}_{m}\}$ among the $M$ workers, the chance that all $M$ attempts fail is simply $p_{f}^{M}$ because $\tau$ forms an i.i.d. process. Therefore, we can see that $M^{\text{th}}$ order diversity gain (increase in the exponent of the failure rate) is achieved at the cost of $M$ times increase in resources. In Figure 4, success rate curves are shown for a benchmark problem. RF-EI($M$=1) is the baseline with success rate equal to $1-p_{f}$, and Theoretical($M$=6) is a synthetic curve calculated from $1-p_{f}^{6}$. Clearly, success rate of Theoretical($M$=6) ramps up much faster than DEEP-BO($M$=1) and demonstrates the power of order six diversity. The exciting part is the performance of DEEP-BO($M$=6) that is shown together. DEEP-BO($M$=6) overwhelmingly outperforms Theoretical($M$=6), indicating that it can achieve much more than order six diversity ($1-p_{f}^{6}$).

A large portion of the substantial extra improvement is due to the mix of $N=6$ individual models with different modeling characteristics, and the effect comes in two-fold. The first is the different (ideally independent) characteristics of the $N$ models where at least one might work well. This is another type of diversity that is different from the $p_{f}^{M}$ effect, and basically says at least one of $p_{f,1},\dots,p_{f,N}$ will likely be small over the $N$ individual models for any given task. The second is the cooperative nature of the diversified algorithm thanks to the sharing of $\{\mathcal{H}_{n}\}$. When an individual model A is not capable of selecting good target points for exploration and is stuck in a bad region, some of the other models might make better selections that serve well as exploratory points to the model A. Then the poorly performing model A can escape from the bad cycle and start to perform well. In our empirical experiments, we extensively tried multi-armed-bandit-like approaches to provide more turns to the better performing individual models, but the strategy actually turned out to hurt the overall performance. It turns out that it is better to give a fair chance to even the worst performing model such that its different modeling characteristic can be utilized. Of course, this is assuming that even the worst performing model is generally a plausible and useful model for DNN HPO.

5.2 Parallelization loss

When the expected time is considered, it would be ideal if the time is reduced by $1/M$ by using $M$ processors. We examined if such a linear gain in performance can be retained as $M$ becomes large. In Figure 4(a), distributions of the expected time are shown for 3, 6, 12, and 18 processors. Expected time and variance can be seen to decrease as the number of processor increases. As for GP-EI, we have found that MCMC(1) performs better probably because the increased randomness is helpful when multiple processors are used. In Figure 4(b), the expected time performance with respect to $\times M$ is shown after normalization. The dotted gray line shows the $\times M$ performance. All the algorithms show the decrease in the normalized performance as $M$ is increased. But the values do not drop very fast after $M$ reaches 4$\sim$5 processors. For parallel diversification, interestingly the performance even went above linear gain when $M$ is between 2 and 4. This is due to the diversification and cooperation.

5.3 Robust termination criterion

Evaluating configurations very early can much save resources, but is possibly hazardous for BO. CR, our ETR, intentionally does not locate $j_{1}$ in early epochs considering cases illustrated in Figure 5 for configurations with "batchnorm" whose termination can prompt false alarms that harm BO. Checkpoint-1($j_{1}$) here was set as rounded $E/2$ to lessen this possibility. Similar reasoning goes for checkpoint-2 whose observation start point for the scope of epochs observed is set as epoch $j_{1}$, not the first epoch. With all other conditions fixed for former-half epochs, the inter-dataset $\mu$ and $\sigma$ of survivor rank regret is strictly counter-proportional to the observation start point for checkpoint-2 of our ETR (see Appendix B for details). These correlations strongly indicate that a delayed observation start point may encompass more reliable information (smaller $\mu$) and allow greater robustness (smaller $\sigma$).

Notable observations with our ETR are DEEP-BO’s results compared to MSR and BOHB in Figure 2 and the comparison of ablation results about "Early Termination" in Table 3. That DEEP-BO outperforms MSR in our benchmarks is not too surprising as our rule was developed on MSR. The more interesting part is that from the perspective of success rates, DEEP-BO outperforms BOHB even though both methods conceptually refer to similar parts of the envelope function (Li et al., 2017).

DEEP-BO abstains from evaluating too early and specifies envelope functions to be more evidently applicable to well-trainable configurations. BOHB evaluates from early epochs which can be dangerous for cases like Figure 5 particularly when all observed best configurations have poor initial performances. Terminating very early does help BO’s explorability and allows the algorithm to reach easy goal performances very rapidly. However, this could increase false alarms, hindering the algorithm from performing beyond such easy targets. Success rate is the average of Boolean results where the iteration is matched with 0 if it does not exceed goal performance. The goal for our study is set high and thereby BOHB could have failed drastically.

Envelope function overlappings should be minimal to be applicable. Though this is more evidently observed with high performing configurations, sporadic fluctuations that can bother evaluation still exist as Figure 6 illustrates. Thereby, it would be crucial for the ETR to be less fluctuation-prone. This can be aided by referring to a range of epochs rather than just a single epoch. Our rule and MSR are designed in such a way and therefore are in advantage in regards of robustness.

Table 3 shows both the strength of DEEP-BO’s ETR and the general limit of ETRs. The fact that DEEP-BO with CR was best for 5 out of 6 cases heuristically supports its robustness. However, one case revealed that there might be no always good ETR. CR was designed to be more robust than its precedents but still has its limits when used alone. Observations recommend ETRs be used along with other methods that help BO’s robustness as this work attempts.

6 Conclusion

In this work, we proposed three different strategies to enhance Bayesian Optimization for DNN HPO with a pursuit of improved average performance and enhanced robustness. We first adopted the concept of diversification. Individual BO models tend to focus on different parts of search space due to the different modeling characteristics. With diversification, each algorithm ends up serving as an excellent exploration guide to the other individual algorithms while being a greedy exploiter for its own selection. Then, we proposed an early termination strategy that works well by being conservative. Learning curves of DNN can show distinct patterns depending on what options (e.g. batchnorm) are turned on, and it is very important to avoid being too aggressive. For parallel processing, we suggested the in-progress method that uses premature evaluation results of pending DNN evaluations. Even when a premature evaluation is inaccurate and harmful, the problem is self-corrected after completing the evaluation. Additionally, we introduced a cost function transformation technique that can be helpful. Overall, the resulting DEEP-BO works very well and outperformed the existing state-of-the-art algorithms for the six benchmark DNN HPO tasks that we have investigated.

Appendix A. Algorithm Description

Hybrid log transformation

[TABLE]

Note that ($\operatorname{log}(\alpha)$ - $\alpha$) is added to make $g(y)$ a continuous function. In our work, we use $\alpha=0.3$. When $f(\mathbf{x})$ is not an accuracy, its range can be scaled to come within [0,1].

Compound termination rule

We designed an early termination rule and named as the Compound Rule (CR). We first introduce two formulaic tools required for computing function $l(E,\beta)$ and $h(j,\mathcal{A},\mathcal{H},\beta)$ which constitute our CR. Function $F_{X}(x)$ refers to the cumulative distribution function (CDF) for the distribution of a given random variable $X$. The rule’s design constrains the range of $\beta$ to $\beta\in{(0.0,\;0.5]}$.

[TABLE]

Next, we show $l$ which returns the two checkpoints’ epoch locations (epoch $j_{1}$ and $j_{2}$) and define $S$, that holds the history for configurations that survived the checkpoint $j_{1}$ computed by function $l$.

[TABLE]

Using the above, we show $h$ which returns one of two thresholds. If $j\leq{j_{1}}$, $h$ returns the threshold for epoch $j_{1}$ (checkpoint-1), and otherwise returns the threshold for epoch $j_{2}$ (checkpoint-2).

[TABLE]

Under a hypothetical environment where mapping early terminated configurations with intermediate performances does not penalize BO’s modeling and per-epoch training time is fixed at $\frac{1}{M}$, larger $\beta$ is observed to increase the number of configurations a BO process expectedly encounters per unit time. This is expressed as $q(\beta)$ below:

[TABLE]

$q(\beta)$ initially increases almost linearly with $\beta$ but starts to wane away from this linear trajectory from about $\beta=0.2$. Though this conditioned observation suggests choosing $\beta\in{[0.05,\;0.20]}$, such condition would not hold for practical BO processes, making deterministic assumptions difficult. $\beta$ may be tuned based on practical user constraints.

Appendix B.  Early Termination Rule (ETR)

Structural components for ETR

ETRs rely on parameters that control several smaller knobs composing such structures. Traditionally, works in ETRs explore the theoretical and empirical robustness of the ETR as a single compound but not these knobs that comprise that compound. Comprehension of these knobs and their inner-workings is likely to further our understanding of early termination and potentially provide intuitions for ETR designs. In this part of our study, we attempt to observe and objectify some common knobs that many ETRs may share.

We can annotate that an ETR evaluates works on a learning curve model $L_{\Theta}$ where $\Theta$ defines $L$ with preset features $\{\theta_{1},\theta_{2}...\theta_{k}\}$ for $k\in\mathbb{N}$. Each $\theta_{k}$ refers to the $L$’s features which include, but are not excluded to, the maximum budget for training a configuration, whether $L$ is a regression model, refers to configuration history for adaptation, or uses extrapolated fantasy values for modeling. $L_{\Theta}$ takes 2 vector inputs: the selected hyperparameter configuration $\mathbf{x}\in\mathbf{X}$ and a design parameter vector $v\in\mathbf{V}$ to decide whether to terminate $\mathbf{x}$. If $\mathbf{T}=\{0,1\}$ where 0 means termination and 1 means no termination,

[TABLE]

whose domain is a Cartesian product. $\mathbf{X}$ would be forced to be an ordered vector space if some $\theta_{k}\in\Theta$ dictates $L_{\Theta}$ to refer to the configuration history.

Number of elements in $v$ vary according to the proposed ETRs but are common in that they control several inner knobs at once. We observe and objectify 4 inner knobs and with them build on this equation.

• •

Observation Start Point $s$: This refers to a point from which the learning curve is meaningfully observed to compute the threshold. $s$ is usually defined as a position relative to the maximum trainable epochs or resources allocated to $\mathbf{x}$. Its value is observed to be proportional to the maximum epochs and is usually expressed as its percentile value.

• •

Observation End Point $e$: This refers to a point until which the learning curve is meaningfully observed to compute the threshold. $e$ is also observed to be defined proportional to the maximum epochs and together with point $s$ determines the scope of learning curve information that the ETR employs for evaluating $\mathbf{x}$. $s$ by definition cannot exceed $e$. Cases where $s=e$ is when the ETR uses the performance at that epoch alone to evaluate $\mathbf{x}$.

• •

Observation Checkpoint $j$: This refers to a point where the learning curve is ultimately evaluated for termination. $j$ is also observed to be defined proportional to the maximum epochs. $e$ by definition cannot exceed $j$. Many cases set $j=e$ so that the evaluation of $\mathbf{x}$ is made right after the end of observation. For our study, we fix $j$ to $e$.

• •

Termination Threshold $h$: This refers to a knob that controls the threshold aggressiveness at the point $e$ with which the configuration is terminated or not. $h\in[0,1]$ where $h=0$ essentially equals to no ETR applied and $h=1$ indiscriminately terminates all configurations at point $e$. Given $v$, $h$ with a vector input of performances defined by $s$ and $e$ determines how the $L_{\Theta}$ should evaluate $\mathbf{x}$ at point $j$.

An ETR that has a single checkpoint to evaluate $\mathbf{x}$ requires one value for each knob. An ETR that has $n$ checkpoints would require ordered vectors for each knob at length $n$ so that checkpoint-$i(i\leq n)$ corresponds to $s_{i}$, $e_{i}$, $j_{i}$, and $h_{i}$.

Survivor Rank Regret

Survivor rank regret is calculated as the mean value of rank regrets for configurations that are not terminated by the ETR. We first explain rank regrets and then survivor rank regret.

Rank regret is an adaption of the ‘immediate regret’ [Shah and Ghahramani, 2015] whose original definition is $r_{t}=|f(\tilde{\mathbf{x}}_{t})-f(\mathbf{x}^{*})|$, where $\mathbf{x}^{*}$ is the known optimizer of $f$ and $\tilde{\mathbf{x}}_{t}$ is the recommendation of an algorithm after $t$ batch evaluations. Rank regret uses ranks as the basic unit of measurement instead of $f(\mathbf{x})$ where the ranks are derived by using the pre-evaluated configuration tables generated with the Sobol sequences. Rank regret is used for our experiment instead because our work attempts to compare between datasets to check for the algorithm’s robustness. ‘Immediate regret’ is not suitable for such inter-dataset analysis because each dataset has different characteristics and different distributions of achievable performance and therefore the scale of $r_{t}$ values about BO vary greatly about datasets. Amount of change observed in $r_{t}$ for before and after applying some ETR to BO is thereby useless for inter-dataset comparison and analysis.

Rank regret, and thereby survivor rank regret, on the other hand is able to deliver information on relative performance improvement. Rank regret first lines up the configurations according to their best performances and ranks the best among those configurations as the first. If we define the rank of $\mathbf{x}$ in the table as $rank(\mathbf{x})$, we normalize its difference with $rank(\mathbf{x}^{*})$ which by definition is 1 so we get Rank regret $=\frac{rank(\mathbf{x})-rank(\mathbf{x}^{*})}{n(data\;table)}$ where $n(data\;table)$ refer to the number of configurations in the table. Our pre-evaluated configurations for each observed dataset contained 20,000 configurations, so $n(data\;table)$ for this work would simply equal 20,000.

Survivor rank regret here is simply the mean value of rank regret values for a subset of observed configurations that were not terminated at any checkpoint of the ETR and was fully trained to the maximum allocated budget for the configuration. This metric is intended to evaluate whether the configurations that the ETR decided to be worth fully training were, on average, better than those chosen without the ETR. Smaller survivor rank regret values would indicate that the surviving pool of configurations are those with high performances and therefore more likely reach some given goal accuracy. If they are not as good as the baseline average rank regret of an BO with no ETR, then that ETR would be of no help, or even harming, the BO. As this metric is intended for inter-dataset comparison, small inter-dataset variance values for respective survivor rank regret values would signify that the ETR of concern is more likely robust in regards of ETR’s contribution to BO.

A Delayed Observation Start Point for CR’s Checkpoint-2

This part of the work incorporates the structural framework introduced in this appendix to observe experiment results summarized in Table B1 and suggest the motivations for choosing a delayed observation start point ($s_{2}=j_{1}=\lfloor{E/2}\rfloor$) for checkpoint-2. We note that the results in this experiment is a full version of CR but closer to MSR particularly intended for CR’s checkpoint-2. This experiment limits the focus to observing the effects of the change in $s_{2}$ on the survivor rank regret which acted as one motivation for the design choice of CR’s checkpoint-2.

As shown in Table B1, $s_{2}\in\{0.0,0.1,0.2,0.3,0.4\}$ and $e_{2}\in\{0.5,0.7,0.9\}$ (those corresponding to checkpoint-1 is $s_{1}$ and $e_{1}$) together form the scope of epochs observed in producing the 90th percentile threshold at $e_{2}=j_{2}$ for checkpoint-2 (subscript 2 indicates that $e$ and $j$ are those corresponding to checkpoint-2). Among these 2 knobs, we wish to focus on $s_{2}$ in particular where for the given column, the inter-dataset $\mu$ and $\sigma$ values for survivor rank regret strictly decrease as $s_{2}$ increases. This work considers this dynamic to be an important marker for using a delayed $s_{2}$ for checkpoint-2 in regards of enhancing both performance and robustness. Observation of decreasing $\mu$ indicates that this algorithm (with a delayed $s_{2}$) improves BO’s potential performance for datasets overall and the observation with decreasing $\sigma$ indicates greater robustness among datasets. Though the dynamic between $s_{2}$ and $\mu$ does contribute to the reasoning for our choice of $s_{2}$ for checkpoint-2, our work puts more emphasis on the dynamic between $s_{2}$ and $\sigma$.

Smaller $\sigma$ among datasets is an indicator that the effects of an algorithm on BO, whether good or bad (here is the former considering the observations with $\mu$ values), is stable and therefore more likely to act in such steady manner even for unseen datasets. This observation with decreasing $\sigma$ can also be intuitively inferred, as exemplified in Figure 4, considering the fact that the overall maximum deviations in performance are smaller (though that amount varies according to the type of curves) for latter epochs. As stated in the paper, occasional disasterous failures are intolerable and thereby the quality of some algorithm to perform well and steadily is crucial for practical DNN HPO. Therefore, the early termination rule for our DEEP-BO algorithm chooses to have a more delayed observation start point. As checkpoint-2 considers only the configurations that survived checkpoint-1 at epoch $j_{1}$, the observation start point for checkpoint-2 here is set at $j_{1}$ for convenience.

ETR performance comparison

Evaluating the ETR performance only, we compare our ETR, which here we name as Compound rule(CR), to existing ETRs such as Median Stopping Rule(MSR) [Golovin et al., 2017], Learning Curve Extrapolation(LCE) [Domhan et al., 2015]. We select GP-EI as a default BO algorithm which will be accelerated. We evaluated BOHB instead of HB. BOHB is an extended version of HB, which works like an ETR by dynamic resource allocation manner, and also this algorithm already utilizes the other BO. Baker et al. [2017] is excluded in this study because this requires a long warm-up time to train the regression model, which is impractical to our benchmark problem.

As shown in Figure B1, our CR performs better in MNIST-LeNet1 or similar in CIFAR10-ResNet than MSR. However, LCE and BOHB which performs similarly with CR in MNIST-LeNet1 perform worst in CIFAR10-ResNet. Especially, the ETRs which performs worse than random means that they failed severely. We tried best to optimize their termination criteria but, we could not make them robust in this benchmark. These model-based algorithms assume that the asymptotic performance of a given configuration observing with just partial points in a learning curve can be highly predictable. This may cause their prediction failures and also modeling failure of BO. However, model-free algorithms such as MSR seem to perform robustly in any benchmark.

Appendix C. Experiment Design

Benchmark algorithms

For the modeling algorithms, we adopted two of the most popular modeling algorithms of BO - GP and RF. Specifically, we have closely followed the implementations of Spearmint for GP and SMAC for RF. For GP, we used the automatic relevance determination(ARD) Matérn 5/2 kernel and a Monte Carlo estimation of the integrated acquisition as suggested in Snoek et al. [2012]. For RF, we used the hyperparameter setting where the number of trees and the number of minimum items in a split are set to 50, 2, respectively. We also one-hot encoded for categorical hyperparameter values to avoid unexpected distortion of modeling.

For the acquisition function, we utilized the three basic approaches - EI, PI, and UCB. By considering all possible combinations between the modeling algorithms and the acquisition functions, we ended up with six different algorithms. For DEEP-BO, we have adopted the six combinations as individual algorithms and implemented the diversification strategies. The algorithms that are evaluated are summarized below.

• •

The random algorithm [Bergstra and Bengio, 2012]

• •

Six individual BO algorithms. Cartesian product by two modeling algorithms [Snoek et al., 2012, Hutter et al., 2011] and three acquisition functions(Probability of Improvement, Expected Improvement, and Upper Confidence Bound) [Kushner, 1964, Mockus et al., 1978, Srinivas et al., 2009]: GP-EI, GP-PI, GP-UCB, RF-EI, RF-PI, and RF-UCB

• •

GP-Hedge introduced by Hoffman et al. [2011], where GP-EI, GP-PI, GP-UCB are used as the three arms of multi-armed bandit

• •

RF-EI using Learning Curve Extrapolation (LCE) [Domhan et al., 2015].

• •

GP-EI using Median Stopping Rule (MSR) [Golovin et al., 2017]

• •

Hyperband (HB) Li et al. [2017]

• •

BOHB which allocates resources as an infinite bandit-based approach but, it samples better than random using TPE [Falkner et al., 2018].

• •

Our method DEEP-BO.

Random algorithm was included because it is frequently used to replace grid-search, but typically its performance is much worse than the others for DNN tasks. This will become obvious in the result plots.

Hyperparameter configuration space