From Regression to Inference: Meta-Learning Predictors for Neural Architecture Search

TL;DR

This paper introduces a meta-learning approach using Convolutional Neural Processes to improve performance prediction in neural architecture search, especially under limited data conditions.

Contribution

It proposes modeling NAS performance prediction as a conditional function inference problem with meta-learning, enhancing generalization from few samples.

Findings

Achieves state-of-the-art architecture selection with limited samples.

Improves top-K ranking quality in NAS benchmarks.

Demonstrates better generalization compared to traditional regression predictors.

Abstract

Prediction-based approaches are widely used in neural architecture search (NAS), where a predictor estimates the performance of candidate architectures to guide selection. However, existing predictors are typically trained via supervised regression on limited samples, leading to overfitting and poor generalization to unseen architectures. In this work, we propose a fundamentally different formulation that models performance prediction as a conditional function inference problem using a Convolutional Neural Process (ConvNP) with meta-learning capabilities. Instead of fitting a fixed mapping to limited samples, our approach meta-learns to infer performance from partial observations by training with context-target splits across a group of synthesized tasks, explicitly optimizing for generalization under data scarcity and aligning the training procedure with the deployment setting in NAS. We further design simple yet effective meta-features for cell-based architectures and evaluate our method on NAS-Bench-101 and NAS-Bench-201. Extensive experiments show that our approach consistently improves top-K ranking quality and achieves the state-of-the-art architecture selection using limited samples.