Traditional and accelerated gradient descent for neural architecture search

TL;DR

This paper introduces two new neural architecture search algorithms, NASGD and NASAGD, based on optimal transport theory, which efficiently explore many architectures with comparable accuracy to existing methods.

Contribution

The paper presents novel gradient-based NAS algorithms leveraging optimal transport geometry, significantly increasing search efficiency over hill climbing methods.

Findings

NASGD on CIFAR-10 achieves 4.06% error in 12 hours on a single GPU.

Algorithms analyze 40 times more architectures than hill climbing methods.

Methods maintain comparable accuracy while reducing computational resources.

Abstract

In this paper we introduce two algorithms for neural architecture search (NASGD and NASAGD) following the theoretical work by two of the authors [5] which used the geometric structure of optimal transport to introduce the conceptual basis for new notions of traditional and accelerated gradient descent algorithms for the optimization of a function on a semi-discrete space. Our algorithms, which use the network morphism framework introduced in [2] as a baseline, can analyze forty times as many architectures as the hill climbing methods [2, 14] while using the same computational resources and time and achieving comparable levels of accuracy. For example, using NASGD on CIFAR-10, our method designs and trains networks with an error rate of 4.06 in only 12 hours on a single GPU.