Understanding the wiring evolution in differentiable neural architecture search
Sirui Xie, Shoukang Hu, Xinjiang Wang, Chunxiao Liu, Jianping Shi,, Xunying Liu, Dahua Lin

TL;DR
This paper investigates how differentiable neural architecture search (NAS) methods evolve wiring topologies, revealing implicit biases and proposing a unified view to better understand their search dynamics and limitations.
Contribution
It introduces a unified framework for analyzing differentiable NAS, providing empirical and theoretical insights into their implicit biases and wiring evolution mechanisms.
Findings
Wider networks are preferred over deeper ones in differentiable NAS.
Search processes tend to grow architectures rather than prune.
No edges are selected during bi-level optimization in existing frameworks.
Abstract
Controversy exists on whether differentiable neural architecture search methods discover wiring topology effectively. To understand how wiring topology evolves, we study the underlying mechanism of several existing differentiable NAS frameworks. Our investigation is motivated by three observed searching patterns of differentiable NAS: 1) they search by growing instead of pruning; 2) wider networks are more preferred than deeper ones; 3) no edges are selected in bi-level optimization. To anatomize these phenomena, we propose a unified view on searching algorithms of existing frameworks, transferring the global optimization to local cost minimization. Based on this reformulation, we conduct empirical and theoretical analyses, revealing implicit inductive biases in the cost's assignment mechanism and evolution dynamics that cause the observed phenomena. These biases indicate strong…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsDomain Adaptation and Few-Shot Learning · Reinforcement Learning in Robotics · Neural Networks and Applications
MethodsDifferentiable Neural Architecture Search
