Efficient Automatic CASH via Rising Bandits

TL;DR

This paper introduces a novel efficient framework for the CASH problem in AutoML by combining alternating optimization with Rising Bandits, significantly improving efficiency and performance over existing Bayesian optimization methods.

Contribution

The paper proposes an alternating optimization framework with Rising Bandits for CASH, reducing hyperparameter space and accelerating algorithm selection in AutoML.

Findings

Outperforms baseline methods on 30 OpenML datasets.

Reduces hyperparameter search space for BO methods.

Provides theoretical guarantees for the Rising Bandits algorithm.

Abstract

The Combined Algorithm Selection and Hyperparameter optimization (CASH) is one of the most fundamental problems in Automatic Machine Learning (AutoML). The existing Bayesian optimization (BO) based solutions turn the CASH problem into a Hyperparameter Optimization (HPO) problem by combining the hyperparameters of all machine learning (ML) algorithms, and use BO methods to solve it. As a result, these methods suffer from the low-efficiency problem due to the huge hyperparameter space in CASH. To alleviate this issue, we propose the alternating optimization framework, where the HPO problem for each ML algorithm and the algorithm selection problem are optimized alternately. In this framework, the BO methods are used to solve the HPO problem for each ML algorithm separately, incorporating a much smaller hyperparameter space for BO methods. Furthermore, we introduce Rising Bandits, a CASH-oriented Multi-Armed Bandits (MAB) variant, to model the algorithm selection in CASH. This framework can take the advantages of both BO in solving the HPO problem with a relatively small hyperparameter space and the MABs in accelerating the algorithm selection. Moreover, we further develop an efficient online algorithm to solve the Rising Bandits with provably theoretical guarantees. The extensive experiments on 30 OpenML datasets demonstrate the superiority of the proposed approach over the competitive baselines.