A resource-efficient method for repeated HPO and NAS problems

TL;DR

This paper introduces a resource-efficient extension of Successive Halving for repeated hyperparameter and neural architecture search, leveraging prior information to reduce costs while maintaining accuracy.

Contribution

It presents a simple, effective transfer learning method for repeated HNAS that outperforms existing approaches in efficiency and robustness.

Findings

Significantly reduces computational costs in repeated HNAS

Maintains high accuracy and robustness to negative transfer

Sets a new baseline for transfer learning in HNAS

Abstract

In this work we consider the problem of repeated hyperparameter and neural architecture search (HNAS). We propose an extension of Successive Halving that is able to leverage information gained in previous HNAS problems with the goal of saving computational resources. We empirically demonstrate that our solution is able to drastically decrease costs while maintaining accuracy and being robust to negative transfer. Our method is significantly simpler than competing transfer learning approaches, setting a new baseline for transfer learning in HNAS.