Towards Robust and Automatic Hyper-Parameter Tunning

TL;DR

This paper introduces autoHyper, a novel hyper-parameter optimization method that uses low-rank convolutional weight factorization to define an analytical response surface, enabling efficient and robust hyper-parameter tuning with less computational cost.

Contribution

The paper proposes a new HPO approach leveraging low-rank weight factorization to create an analytical response surface, improving efficiency and generalization over existing methods.

Findings

autoHyper outperforms Bayesian Optimization in experiments

The method generalizes across models, optimizers, and datasets

It reduces computational costs of hyper-parameter tuning

Abstract

The task of hyper-parameter optimization (HPO) is burdened with heavy computational costs due to the intractability of optimizing both a model's weights and its hyper-parameters simultaneously. In this work, we introduce a new class of HPO method and explore how the low-rank factorization of the convolutional weights of intermediate layers of a convolutional neural network can be used to define an analytical response surface for optimizing hyper-parameters, using only training data. We quantify how this surface behaves as a surrogate to model performance and can be solved using a trust-region search algorithm, which we call autoHyper. The algorithm outperforms state-of-the-art such as Bayesian Optimization and generalizes across model, optimizer, and dataset selection. Our code can be found at \url{https://github.com/MathieuTuli/autoHyper}.