From Hand-Crafted Metrics to Evolved Training-Free Performance Predictors for Neural Architecture Search via Genetic Programming

TL;DR

This paper introduces a genetic programming-based framework to automatically design zero-cost metrics for neural architecture search, outperforming manual metrics and enabling rapid, high-quality NAS with minimal computational resources.

Contribution

The study presents an automated, extensible method to generate zero-cost proxies for NAS, improving over hand-crafted metrics and reducing search time.

Findings

Automatically generated proxies outperform hand-crafted metrics across diverse NAS problems.

The evolved proxy enables finding competitive architectures within 15 minutes on a single GPU.

The framework is highly extensible and adaptable to different NAS search spaces.

Abstract

Estimating the network performance using zero-cost (ZC) metrics has proven both its efficiency and efficacy in Neural Architecture Search (NAS). However, a notable limitation of most ZC proxies is their inconsistency, as reflected by the substantial variation in their performance across different problems. Furthermore, the design of existing ZC metrics is manual, involving a time-consuming trial-and-error process that requires substantial domain expertise. These challenges raise two critical questions: (1) Can we automate the design of ZC metrics? and (2) Can we utilize the existing hand-crafted ZC metrics to synthesize a more generalizable one? In this study, we propose a framework based on Symbolic Regression via Genetic Programming to automate the design of ZC metrics. Our framework is not only highly extensible but also capable of quickly producing a ZC metric with a strong positive rank correlation to true network performance across diverse NAS search spaces and tasks. Extensive experiments on 13 problems from NAS-Bench-Suite-Zero demonstrate that our automatically generated proxies consistently outperform hand-crafted alternatives. Using our evolved proxy metric as the search objective in an evolutionary algorithm, we could identify network architectures with competitive performance within 15 minutes using a single consumer GPU.