Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Search
Xiangxiang Chu, Tianbao Zhou, Bo Zhang, Jixiang Li

TL;DR
Fair DARTS introduces a collaborative approach to neural architecture search that addresses performance collapse caused by unfair competition, achieving state-of-the-art results on CIFAR-10 and ImageNet.
Contribution
The paper proposes a novel fair competition framework in DARTS, relaxing exclusive competition and introducing a zero-one loss for better discretization.
Findings
Eliminates performance collapse in DARTS
Achieves new state-of-the-art results on CIFAR-10
Demonstrates effectiveness across mainstream search spaces
Abstract
Differentiable Architecture Search (DARTS) is now a widely disseminated weight-sharing neural architecture search method. However, it suffers from well-known performance collapse due to an inevitable aggregation of skip connections. In this paper, we first disclose that its root cause lies in an unfair advantage in exclusive competition. Through experiments, we show that if either of two conditions is broken, the collapse disappears. Thereby, we present a novel approach called Fair DARTS where the exclusive competition is relaxed to be collaborative. Specifically, we let each operation's architectural weight be independent of others. Yet there is still an important issue of discretization discrepancy. We then propose a zero-one loss to push architectural weights towards zero or one, which approximates an expected multi-hot solution. Our experiments are performed on two mainstream search…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Neural Network Applications · Domain Adaptation and Few-Shot Learning · Advanced Image and Video Retrieval Techniques
MethodsDifferentiable Architecture Search · Sigmoid Activation · Tanh Activation · Softmax · Long Short-Term Memory
