Hyperparameter Optimization in Neural Networks via Structured Sparse Recovery

TL;DR

This paper introduces a structured sparse recovery framework for hyperparameter optimization and neural architecture search, demonstrating improved performance and efficiency through novel algorithms and theoretical analysis.

Contribution

It establishes a new connection between HPO/NAS and structured sparse recovery, proposing algorithms that enhance search efficiency and discovering new neural architectures.

Findings

Improved hyperparameter optimization performance on CIFAR-10.

Proposed CoNAS algorithm outperforms existing NAS methods.

Theoretical bounds on validation error measurements for NAS.

Abstract

In this paper, we study two important problems in the automated design of neural networks -- Hyper-parameter Optimization (HPO), and Neural Architecture Search (NAS) -- through the lens of sparse recovery methods. In the first part of this paper, we establish a novel connection between HPO and structured sparse recovery. In particular, we show that a special encoding of the hyperparameter space enables a natural group-sparse recovery formulation, which when coupled with HyperBand (a multi-armed bandit strategy), leads to improvement over existing hyperparameter optimization methods. Experimental results on image datasets such as CIFAR-10 confirm the benefits of our approach. In the second part of this paper, we establish a connection between NAS and structured sparse recovery. Building upon ``one-shot'' approaches in NAS, we propose a novel algorithm that we call CoNAS by merging ideas from one-shot approaches with a techniques for learning low-degree sparse Boolean polynomials. We provide theoretical analysis on the number of validation error measurements. Finally, we validate our approach on several datasets and discover novel architectures hitherto unreported, achieving competitive (or better) results in both performance and search time compared to the existing NAS approaches.