An Empirical Study on Ensemble-Based Transfer Learning Bayesian Optimisation with Mixed Variable Types

TL;DR

This paper empirically analyzes ensemble-based transfer learning methods in Bayesian optimisation, introducing new pipeline components and benchmarks, and finds warm start initialization and positive weight constraints improve performance.

Contribution

It introduces specific pipeline components, including a weighting strategy for ensemble models and benchmarks, advancing transfer learning in Bayesian optimisation.

Findings

Warm start initialization enhances transfer learning performance.

Constraining ensemble weights to be positive improves results.

New real-time benchmarks facilitate future research.

Abstract

Bayesian optimisation is a sample efficient method for finding a global optimum of expensive black-box objective functions. Historic datasets from related problems can be exploited to help improve performance of Bayesian optimisation by adapting transfer learning methods to various components of the Bayesian optimisation pipeline. In this study we perform an empirical analysis of various ensemble-based transfer learning Bayesian optimisation methods and pipeline components. We expand on previous work in the literature by contributing some specific pipeline components, and three new real-time transfer learning Bayesian optimisation benchmarks. In particular we propose to use a weighting strategy for ensemble surrogate model predictions based on regularised regression with weights constrained to be positive, and a related component for handling the case when transfer learning is not improving Bayesian optimisation performance. We find that in general, two components that help improve transfer learning Bayesian optimisation performance are warm start initialisation and constraining weights used with ensemble surrogate model to be positive.