Sequential Model-Based Ensemble Optimization

TL;DR

This paper introduces an extension to sequential model-based optimization that automatically constructs ensembles of models, improving performance over single-model selection in machine learning tasks.

Contribution

It proposes a novel SMBO extension leveraging agnostic Bayesian learning to automatically build ensembles, enhancing model performance.

Findings

Ensembles constructed via the proposed method outperform single models selected by SMBO.

The approach is effective across diverse regression and classification datasets.

Experimental results confirm the superiority of ensemble optimization over traditional model selection.

Abstract

One of the most tedious tasks in the application of machine learning is model selection, i.e. hyperparameter selection. Fortunately, recent progress has been made in the automation of this process, through the use of sequential model-based optimization (SMBO) methods. This can be used to optimize a cross-validation performance of a learning algorithm over the value of its hyperparameters. However, it is well known that ensembles of learned models almost consistently outperform a single model, even if properly selected. In this paper, we thus propose an extension of SMBO methods that automatically constructs such ensembles. This method builds on a recently proposed ensemble construction paradigm known as agnostic Bayesian learning. In experiments on 22 regression and 39 classification data sets, we confirm the success of this proposed approach, which is able to outperform model selection with SMBO.