Discretization-Aware Architecture Search

TL;DR

This paper introduces discretization-aware architecture search (DA2S), a method that reduces accuracy loss during neural architecture discretization by guiding the super-network towards desired topologies, improving NAS results especially in imbalanced configurations.

Contribution

The paper proposes a novel loss term in NAS that mitigates discretization errors, enhancing architecture quality in imbalanced network configurations.

Findings

DA2S outperforms existing NAS methods on standard benchmarks.

The approach significantly reduces accuracy loss during discretization.

Effective in imbalanced network configurations.

Abstract

The search cost of neural architecture search (NAS) has been largely reduced by weight-sharing methods. These methods optimize a super-network with all possible edges and operations, and determine the optimal sub-network by discretization, \textit{i.e.}, pruning off weak candidates. The discretization process, performed on either operations or edges, incurs significant inaccuracy and thus the quality of the final architecture is not guaranteed. This paper presents discretization-aware architecture search (DA\textsuperscript{2}S), with the core idea being adding a loss term to push the super-network towards the configuration of desired topology, so that the accuracy loss brought by discretization is largely alleviated. Experiments on standard image classification benchmarks demonstrate the superiority of our approach, in particular, under imbalanced target network configurations that were not studied before.