CAP: A Context-Aware Neural Predictor for NAS

TL;DR

This paper introduces CAP, a context-aware neural predictor that efficiently predicts neural architecture performance using minimal annotated data by leveraging architecture graphs and self-supervised learning, significantly improving NAS efficiency.

Contribution

The paper presents a novel context-aware neural predictor that requires fewer annotated architectures by encoding architectures as graphs and using self-supervised learning, enhancing NAS performance.

Findings

CAP achieves precise architecture ranking with only 172 annotations.

CAP outperforms state-of-the-art neural predictors in various search spaces.

CAP effectively finds promising architectures in NAS-Bench-101 and DARTS spaces.

Abstract

Neural predictors are effective in boosting the time-consuming performance evaluation stage in neural architecture search (NAS), owing to their direct estimation of unseen architectures. Despite the effectiveness, training a powerful neural predictor with fewer annotated architectures remains a huge challenge. In this paper, we propose a context-aware neural predictor (CAP) which only needs a few annotated architectures for training based on the contextual information from the architectures. Specifically, the input architectures are encoded into graphs and the predictor infers the contextual structure around the nodes inside each graph. Then, enhanced by the proposed context-aware self-supervised task, the pre-trained predictor can obtain expressive and generalizable representations of architectures. Therefore, only a few annotated architectures are sufficient for training. Experimental results in different search spaces demonstrate the superior performance of CAP compared with state-of-the-art neural predictors. In particular, CAP can rank architectures precisely at the budget of only 172 annotated architectures in NAS-Bench-101. Moreover, CAP can help find promising architectures in both NAS-Bench-101 and DARTS search spaces on the CIFAR-10 dataset, serving as a useful navigator for NAS to explore the search space efficiently.