Successive Halving with Learning Curve Prediction via Latent Kronecker Gaussian Processes

TL;DR

This paper explores enhancing the Successive Halving hyperparameter optimization algorithm by integrating learning curve predictions using Latent Kronecker Gaussian Processes, aiming to better identify promising candidates and avoid premature pruning.

Contribution

The study introduces a predictive approach to Successive Halving that utilizes Latent Kronecker Gaussian Processes for learning curve prediction, addressing limitations of current performance-based decisions.

Findings

Predictive approach achieves competitive performance.

Requires fully observed learning curves for training.

Potential improvements if existing learning curve data is leveraged.

Abstract

Successive Halving is a popular algorithm for hyperparameter optimization which allocates exponentially more resources to promising candidates. However, the algorithm typically relies on intermediate performance values to make resource allocation decisions, which can cause it to prematurely prune slow starters that would eventually become the best candidate. We investigate whether guiding Successive Halving with learning curve predictions based on Latent Kronecker Gaussian Processes can overcome this limitation. In a large-scale empirical study involving different neural network architectures and a click prediction dataset, we compare this predictive approach to the standard approach based on current performance values. Our experiments show that, although the predictive approach achieves competitive performance, it is not Pareto optimal compared to investing more resources into the standard approach, because it requires fully observed learning curves as training data. However, this downside could be mitigated by leveraging existing learning curve data.