Modeling All Response Surfaces in One for Conditional Search Spaces

TL;DR

This paper introduces a novel Bayesian Optimization method that models all response surfaces in a single framework using self-attention, effectively handling dependencies in conditional hyperparameter spaces for improved AutoML performance.

Contribution

It proposes a structure-aware hyperparameter embedding and an attention-based feature extractor to unify modeling of multiple subspaces with a single Gaussian Process, capturing inter-subspace relationships.

Findings

Improves BO efficiency in conditional search spaces.

Demonstrates superior performance on real-world tasks.

Outperforms existing methods on HPO-B benchmark.

Abstract

Bayesian Optimization (BO) is a sample-efficient black-box optimizer commonly used in search spaces where hyperparameters are independent. However, in many practical AutoML scenarios, there will be dependencies among hyperparameters, forming a conditional search space, which can be partitioned into structurally distinct subspaces. The structure and dimensionality of hyperparameter configurations vary across these subspaces, challenging the application of BO. Some previous BO works have proposed solutions to develop multiple Gaussian Process models in these subspaces. However, these approaches tend to be inefficient as they require a substantial number of observations to guarantee each GP's performance and cannot capture relationships between hyperparameters across different subspaces. To address these issues, this paper proposes a novel approach to model the response surfaces of all subspaces in one, which can model the relationships between hyperparameters elegantly via a self-attention mechanism. Concretely, we design a structure-aware hyperparameter embedding to preserve the structural information. Then, we introduce an attention-based deep feature extractor, capable of projecting configurations with different structures from various subspaces into a unified feature space, where the response surfaces can be formulated using a single standard Gaussian Process. The empirical results on a simulation function, various real-world tasks, and HPO-B benchmark demonstrate that our proposed approach improves the efficacy and efficiency of BO within conditional search spaces.