Multi-Objective Neural Architecture Search by Learning Search Space Partitions

TL;DR

This paper introduces a novel multi-objective neural architecture search method using LaMOO, significantly improving search efficiency and achieving competitive accuracy with fewer samples and lower computational costs.

Contribution

The work applies LaMOO to NAS, enabling efficient multi-objective optimization by learning search space partitions, resulting in over 200% sample efficiency gains.

Findings

Over 200% improvement in sample efficiency compared to Bayesian and evolutionary methods.

Achieved 97.36% accuracy on CIFAR10 with only 1.62M parameters.

Reaches 80.4% top-1 accuracy on ImageNet with 522M FLOPs.

Abstract

Deploying deep learning models requires taking into consideration neural network metrics such as model size, inference latency, and #FLOPs, aside from inference accuracy. This results in deep learning model designers leveraging multi-objective optimization to design effective deep neural networks in multiple criteria. However, applying multi-objective optimizations to neural architecture search (NAS) is nontrivial because NAS tasks usually have a huge search space, along with a non-negligible searching cost. This requires effective multi-objective search algorithms to alleviate the GPU costs. In this work, we implement a novel multi-objectives optimizer based on a recently proposed meta-algorithm called LaMOO on NAS tasks. In a nutshell, LaMOO speedups the search process by learning a model from observed samples to partition the search space and then focusing on promising regions likely to contain a subset of the Pareto frontier. Using LaMOO, we observe an improvement of more than 200% sample efficiency compared to Bayesian optimization and evolutionary-based multi-objective optimizers on different NAS datasets. For example, when combined with LaMOO, qEHVI achieves a 225% improvement in sample efficiency compared to using qEHVI alone in NasBench201. For real-world tasks, LaMOO achieves 97.36% accuracy with only 1.62M #Params on CIFAR10 in only 600 search samples. On ImageNet, our large model reaches 80.4% top-1 accuracy with only 522M #FLOPs.