Obeying the Order: Introducing Ordered Transfer Hyperparameter Optimisation

TL;DR

This paper introduces ordered transfer hyperparameter optimisation (OTHPO), a method that leverages task order in sequential hyperparameter tuning, outperforming existing transfer HPO methods in various benchmarks.

Contribution

It formally defines OTHPO, differentiates it from related problems, and demonstrates its superior performance over state-of-the-art transfer HPO methods.

Findings

OTHPO outperforms existing transfer HPO methods in benchmarks.

Order-aware approach improves hyperparameter tuning in sequential tasks.

Open-sourced benchmarks facilitate future research on ordered transfer HPO.

Abstract

We introduce ordered transfer hyperparameter optimisation (OTHPO), a version of transfer learning for hyperparameter optimisation (HPO) where the tasks follow a sequential order. Unlike for state-of-the-art transfer HPO, the assumption is that each task is most correlated to those immediately before it. This matches many deployed settings, where hyperparameters are retuned as more data is collected; for instance tuning a sequence of movie recommendation systems as more movies and ratings are added. We propose a formal definition, outline the differences to related problems and propose a basic OTHPO method that outperforms state-of-the-art transfer HPO. We empirically show the importance of taking order into account using ten benchmarks. The benchmarks are in the setting of gradually accumulating data, and span XGBoost, random forest, approximate k-nearest neighbor, elastic net, support vector machines and a separate real-world motivated optimisation problem. We open source the benchmarks to foster future research on ordered transfer HPO.