Raiders of the Lost Architecture: Kernels for Bayesian Optimization in Conditional Parameter Spaces

TL;DR

This paper introduces a new kernel for Bayesian optimization that effectively handles structures with varying parameters, such as neural network architectures with different layers, improving model accuracy and optimization outcomes.

Contribution

The paper presents a novel kernel designed for conditional parameter spaces, enabling better modeling and optimization across structures with differing parameters.

Findings

The new kernel outperforms baseline kernels in model quality.

Bayesian optimization results are improved using the proposed kernel.

The approach effectively handles structures with varying parameter relevance.

Abstract

In practical Bayesian optimization, we must often search over structures with differing numbers of parameters. For instance, we may wish to search over neural network architectures with an unknown number of layers. To relate performance data gathered for different architectures, we define a new kernel for conditional parameter spaces that explicitly includes information about which parameters are relevant in a given structure. We show that this kernel improves model quality and Bayesian optimization results over several simpler baseline kernels.