Neural Architecture Search using Covariance Matrix Adaptation Evolution Strategy

TL;DR

This paper introduces CMANAS, a neural architecture search framework using CMA-ES that significantly reduces search time while achieving high accuracy on CIFAR datasets and effective transfer to ImageNet.

Contribution

The paper presents a novel CMA-ES based framework for neural architecture search that improves efficiency and accuracy over previous evolution-based methods.

Findings

Achieved 97.44% accuracy on CIFAR-10 in 0.45 GPU days.

Achieved 83.24% accuracy on CIFAR-100 in 0.6 GPU days.

Transferred architectures to ImageNet with over 92% top-5 accuracy.

Abstract

Evolution-based neural architecture search requires high computational resources, resulting in long search time. In this work, we propose a framework of applying the Covariance Matrix Adaptation Evolution Strategy (CMA-ES) to the neural architecture search problem called CMANAS, which achieves better results than previous evolution-based methods while reducing the search time significantly. The architectures are modelled using a normal distribution, which is updated using CMA-ES based on the fitness of the sampled population. We used the accuracy of a trained one shot model (OSM) on the validation data as a prediction of the fitness of an individual architecture to reduce the search time. We also used an architecture-fitness table (AF table) for keeping record of the already evaluated architecture, thus further reducing the search time. CMANAS finished the architecture search on CIFAR-10 with the top-1 test accuracy of 97.44% in 0.45 GPU day and on CIFAR-100 with the top-1 test accuracy of 83.24% for 0.6 GPU day on a single GPU. The top architectures from the searches on CIFAR-10 and CIFAR-100 were then transferred to ImageNet, achieving the top-5 accuracy of 92.6% and 92.1%, respectively.