Tune My Adam, Please!

TL;DR

This paper introduces Adam-PFN, a pre-trained surrogate model combined with a novel augmentation method, CDF-augment, to enhance hyperparameter tuning of Adam optimizer using freeze-thaw Bayesian Optimization, especially on out-of-distribution tasks.

Contribution

It presents Adam-PFN, a surrogate model trained on learning curves, and CDF-augment, a data augmentation technique, improving hyperparameter tuning efficiency and accuracy for Adam.

Findings

Enhanced learning curve extrapolation accuracy.

Accelerated hyperparameter optimization process.

Strong performance on out-of-distribution tasks.

Abstract

The Adam optimizer remains one of the most widely used optimizers in deep learning, and effectively tuning its hyperparameters is key to optimizing performance. However, tuning can be tedious and costly. Freeze-thaw Bayesian Optimization (BO) is a recent promising approach for low-budget hyperparameter tuning, but is limited by generic surrogates without prior knowledge of how hyperparameters affect learning. We propose Adam-PFN, a new surrogate model for Freeze-thaw BO of Adam's hyperparameters, pre-trained on learning curves from TaskSet, together with a new learning curve augmentation method, CDF-augment, which artificially increases the number of available training examples. Our approach improves both learning curve extrapolation and accelerates hyperparameter optimization on TaskSet evaluation tasks, with strong performance on out-of-distribution (OOD) tasks.