Warm Starting CMA-ES for Hyperparameter Optimization

TL;DR

This paper introduces WS-CMA-ES, a warm-started CMA-ES method for hyperparameter optimization that leverages prior knowledge to reduce adaptation time and outperform existing approaches, especially under limited evaluation budgets.

Contribution

The paper proposes a novel warm-start CMA-ES method that incorporates prior knowledge through a new task similarity measure, improving hyperparameter optimization efficiency.

Findings

WS-CMA-ES outperforms original CMA-ES and Bayesian optimization in limited evaluation scenarios.

The task similarity measure effectively correlates performance across synthetic problems.

Prior knowledge transfer significantly shortens adaptation time in hyperparameter optimization.

Abstract

Hyperparameter optimization (HPO), formulated as black-box optimization (BBO), is recognized as essential for automation and high performance of machine learning approaches. The CMA-ES is a promising BBO approach with a high degree of parallelism, and has been applied to HPO tasks, often under parallel implementation, and shown superior performance to other approaches including Bayesian optimization (BO). However, if the budget of hyperparameter evaluations is severely limited, which is often the case for end users who do not deserve parallel computing, the CMA-ES exhausts the budget without improving the performance due to its long adaptation phase, resulting in being outperformed by BO approaches. To address this issue, we propose to transfer prior knowledge on similar HPO tasks through the initialization of the CMA-ES, leading to significantly shortening the adaptation time. The knowledge transfer is designed based on the novel definition of task similarity, with which the correlation of the performance of the proposed approach is confirmed on synthetic problems. The proposed warm starting CMA-ES, called WS-CMA-ES, is applied to different HPO tasks where some prior knowledge is available, showing its superior performance over the original CMA-ES as well as BO approaches with or without using the prior knowledge.