Dynamic Ensemble of Low-fidelity Experts: Mitigating NAS "Cold-Start"
Junbo Zhao, Xuefei Ning, Enshu Liu, Binxin Ru, Zixuan Zhou, Tianchen, Zhao, Chen Chen, Jiajin Zhang, Qingmin Liao, Yu Wang

TL;DR
This paper introduces a dynamic ensemble predictor framework that leverages multiple low-fidelity estimations to mitigate the cold-start problem in predictor-based Neural Architecture Search, improving prediction accuracy with less data.
Contribution
It proposes a novel two-step method to fuse diverse low-fidelity information using specialized experts and a gating network, enhancing NAS efficiency.
Findings
Effective across five search spaces with different encoders
Improves prediction accuracy with limited high-fidelity data
Easily integrated into existing NAS frameworks
Abstract
Predictor-based Neural Architecture Search (NAS) employs an architecture performance predictor to improve the sample efficiency. However, predictor-based NAS suffers from the severe ``cold-start'' problem, since a large amount of architecture-performance data is required to get a working predictor. In this paper, we focus on exploiting information in cheaper-to-obtain performance estimations (i.e., low-fidelity information) to mitigate the large data requirements of predictor training. Despite the intuitiveness of this idea, we observe that using inappropriate low-fidelity information even damages the prediction ability and different search spaces have different preferences for low-fidelity information types. To solve the problem and better fuse beneficial information provided by different types of low-fidelity information, we propose a novel dynamic ensemble predictor framework that…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
Taxonomy
TopicsAnomaly Detection Techniques and Applications · Adversarial Robustness in Machine Learning · Software Engineering Research
