Mutually-aware Sub-Graphs Differentiable Architecture Search

TL;DR

This paper introduces MSG-DAS, a novel differentiable architecture search method that combines multi-path and single-path paradigms using mutually exclusive sub-graphs, improving efficiency and performance balance.

Contribution

It proposes a simple, efficient framework with a Gumbel-TopK sampler, Dropblock-Identity stabilization, and super-net guidance distillation to enhance NAS.

Findings

Achieves competitive accuracy on ImageNet and CIFAR10

Balances memory efficiency and search quality effectively

Outperforms or matches recent NAS approaches in performance

Abstract

Differentiable architecture search is prevalent in the field of NAS because of its simplicity and efficiency, where two paradigms, multi-path algorithms and single-path methods, are dominated. Multi-path framework (e.g. DARTS) is intuitive but suffers from memory usage and training collapse. Single-path methods (e.g.GDAS and ProxylessNAS) mitigate the memory issue and shrink the gap between searching and evaluation but sacrifice the performance. In this paper, we propose a conceptually simple yet efficient method to bridge these two paradigms, referred as Mutually-aware Sub-Graphs Differentiable Architecture Search (MSG-DAS). The core of our framework is a differentiable Gumbel-TopK sampler that produces multiple mutually exclusive single-path sub-graphs. To alleviate the severer skip-connect issue brought by multiple sub-graphs setting, we propose a Dropblock-Identity module to stabilize the optimization. To make best use of the available models (super-net and sub-graphs), we introduce a memory-efficient super-net guidance distillation to improve training. The proposed framework strikes a balance between flexible memory usage and searching quality. We demonstrate the effectiveness of our methods on ImageNet and CIFAR10, where the searched models show a comparable performance as the most recent approaches.