Mutually-aware Sub-Graphs Differentiable Architecture Search
Haoxian Tan, Sheng Guo, Yujie Zhong, Matthew R. Scott, Weilin Huang

TL;DR
This paper introduces MSG-DAS, a novel differentiable architecture search method that combines multi-path and single-path paradigms using mutually exclusive sub-graphs, improving efficiency and performance balance.
Contribution
It proposes a simple, efficient framework with a Gumbel-TopK sampler, Dropblock-Identity stabilization, and super-net guidance distillation to enhance NAS.
Findings
Achieves competitive accuracy on ImageNet and CIFAR10
Balances memory efficiency and search quality effectively
Outperforms or matches recent NAS approaches in performance
Abstract
Differentiable architecture search is prevalent in the field of NAS because of its simplicity and efficiency, where two paradigms, multi-path algorithms and single-path methods, are dominated. Multi-path framework (e.g. DARTS) is intuitive but suffers from memory usage and training collapse. Single-path methods (e.g.GDAS and ProxylessNAS) mitigate the memory issue and shrink the gap between searching and evaluation but sacrifice the performance. In this paper, we propose a conceptually simple yet efficient method to bridge these two paradigms, referred as Mutually-aware Sub-Graphs Differentiable Architecture Search (MSG-DAS). The core of our framework is a differentiable Gumbel-TopK sampler that produces multiple mutually exclusive single-path sub-graphs. To alleviate the severer skip-connect issue brought by multiple sub-graphs setting, we propose a Dropblock-Identity module to…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Neural Network Applications · Advanced Image and Video Retrieval Techniques · Domain Adaptation and Few-Shot Learning
