RD-NAS: Enhancing One-shot Supernet Ranking Ability via Ranking Distillation from Zero-cost Proxies
Peijie Dong, Xin Niu, Lujun Li, Zhiliang Tian, Xiaodong Wang, Zimian, Wei, Hengyue Pan, Dongsheng Li

TL;DR
RD-NAS introduces a ranking distillation method that leverages zero-cost proxies to significantly improve the ranking accuracy of one-shot neural architecture search, reducing computational costs and enhancing search effectiveness.
Contribution
The paper proposes a novel ranking distillation approach using zero-cost proxies and margin subnet sampling to improve one-shot NAS ranking consistency.
Findings
Achieves 10.7% improvement in ranking ability on NAS-Bench-201.
Achieves 9.65% improvement in ranking ability on ResNet-based search space.
Demonstrates effectiveness of ranking distillation in neural architecture search.
Abstract
Neural architecture search (NAS) has made tremendous progress in the automatic design of effective neural network structures but suffers from a heavy computational burden. One-shot NAS significantly alleviates the burden through weight sharing and improves computational efficiency. Zero-shot NAS further reduces the cost by predicting the performance of the network from its initial state, which conducts no training. Both methods aim to distinguish between "good" and "bad" architectures, i.e., ranking consistency of predicted and true performance. In this paper, we propose Ranking Distillation one-shot NAS (RD-NAS) to enhance ranking consistency, which utilizes zero-cost proxies as the cheap teacher and adopts the margin ranking loss to distill the ranking knowledge. Specifically, we propose a margin subnet sampler to distill the ranking knowledge from zero-shot NAS to one-shot NAS by…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Neural Network Applications · Domain Adaptation and Few-Shot Learning · Machine Learning and Data Classification
