Reducing The Search Space For Hyperparameter Optimization Using Group Sparsity

TL;DR

This paper introduces a novel hyperparameter optimization algorithm that combines group-sparse recovery with HyperBand, improving search efficiency and performance over existing methods, validated on image datasets like CIFAR-10.

Contribution

It presents a new spectral hyperparameter selection method using group sparsity, enhancing hyperparameter search by integrating with HyperBand.

Findings

Improved hyperparameter optimization performance over Successive Halving and Random Search.

Effective on image datasets such as CIFAR-10.

Demonstrates the benefits of group-sparse recovery in hyperparameter tuning.

Abstract

We propose a new algorithm for hyperparameter selection in machine learning algorithms. The algorithm is a novel modification of Harmonica, a spectral hyperparameter selection approach using sparse recovery methods. In particular, we show that a special encoding of hyperparameter space enables a natural group-sparse recovery formulation, which when coupled with HyperBand (a multi-armed bandit strategy) leads to improvement over existing hyperparameter optimization methods such as Successive Halving and Random Search. Experimental results on image datasets such as CIFAR-10 confirm the benefits of our approach.