Bayesian Optimization for Simultaneous Selection of Machine Learning Algorithms and Hyperparameters on Shared Latent Space

TL;DR

This paper introduces a Bayesian optimization method that embeds multiple ML algorithm hyper-parameter spaces into a shared latent space, enabling more efficient simultaneous algorithm and hyper-parameter selection with fewer observations.

Contribution

The study proposes a novel shared latent space embedding for hyper-parameter spaces, combined with a multi-task surrogate model and adversarial pre-training, improving optimization efficiency.

Findings

Effective on OpenML datasets

Reduces number of observations needed

Improves hyper-parameter and algorithm selection

Abstract

Selecting the optimal combination of a machine learning (ML) algorithm and its hyper-parameters is crucial for the development of high-performance ML systems. However, since the combination of ML algorithms and hyper-parameters is enormous, the exhaustive validation requires a significant amount of time. Many existing studies use Bayesian optimization (BO) for accelerating the search. On the other hand, a significant difficulty is that, in general, there exists a different hyper-parameter space for each one of candidate ML algorithms. BO-based approaches typically build a surrogate model independently for each hyper-parameter space, by which sufficient observations are required for all candidate ML algorithms. In this study, our proposed method embeds different hyper-parameter spaces into a shared latent space, in which a surrogate multi-task model for BO is estimated. This approach can share information of observations from different ML algorithms by which efficient optimization is expected with a smaller number of total observations. We further propose the pre-training of the latent space embedding with an adversarial regularization, and a ranking model for selecting an effective pre-trained embedding for a given target dataset. Our empirical study demonstrates effectiveness of the proposed method through datasets from OpenML.