Transferable Neural Processes for Hyperparameter Optimization

TL;DR

This paper introduces Transfer Neural Processes, a novel hyperparameter optimization method that leverages transfer learning to significantly reduce the number of trials needed, outperforming existing methods on multiple datasets.

Contribution

The paper presents a new transfer learning-based HPO algorithm called Transfer Neural Processes that efficiently utilizes historical trials to accelerate hyperparameter tuning.

Findings

Achieves state-of-the-art performance with fewer trials

Effective transfer learning from historical HPO data

Significantly reduces hyperparameter tuning time

Abstract

Automated machine learning aims to automate the whole process of machine learning, including model configuration. In this paper, we focus on automated hyperparameter optimization (HPO) based on sequential model-based optimization (SMBO). Though conventional SMBO algorithms work well when abundant HPO trials are available, they are far from satisfactory in practical applications where a trial on a huge dataset may be so costly that an optimal hyperparameter configuration is expected to return in as few trials as possible. Observing that human experts draw on their expertise in a machine learning model by trying configurations that once performed well on other datasets, we are inspired to speed up HPO by transferring knowledge from historical HPO trials on other datasets. We propose an end-to-end and efficient HPO algorithm named as Transfer Neural Processes (TNP), which achieves transfer learning by incorporating trials on other datasets, initializing the model with well-generalized parameters, and learning an initial set of hyperparameters to evaluate. Experiments on extensive OpenML datasets and three computer vision datasets show that the proposed model can achieve state-of-the-art performance in at least one order of magnitude less trials.