Enhanced Gradient for Differentiable Architecture Search

TL;DR

This paper introduces an enhanced gradient-based neural architecture search method that optimizes for both accuracy and efficiency, producing compact networks that outperform hand-crafted models in image classification tasks.

Contribution

The paper presents a novel two-stage NAS framework using an enhanced gradient relaxation and evolutionary multi-objective optimization for efficient, high-performance network design.

Findings

Outperforms hand-crafted networks on CIFAR10 and CIFAR100

Achieves high accuracy with networks under one megabit

Significantly reduces network parameter count compared to other NAS methods

Abstract

In recent years, neural architecture search (NAS) methods have been proposed for the automatic generation of task-oriented network architecture in image classification. However, the architectures obtained by existing NAS approaches are optimized only for classification performance and do not adapt to devices with limited computational resources. To address this challenge, we propose a neural network architecture search algorithm aiming to simultaneously improve network performance (e.g., classification accuracy) and reduce network complexity. The proposed framework automatically builds the network architecture at two stages: block-level search and network-level search. At the stage of block-level search, a relaxation method based on the gradient is proposed, using an enhanced gradient to design high-performance and low-complexity blocks. At the stage of network-level search, we apply an evolutionary multi-objective algorithm to complete the automatic design from blocks to the target network. The experiment results demonstrate that our method outperforms all evaluated hand-crafted networks in image classification, with an error rate of on CIFAR10 and an error rate of on CIFAR100, both at network parameter size less than one megabit. Moreover, compared with other neural architecture search methods, our method offers a tremendous reduction in designed network architecture parameters.