GreenFactory: Ensembling Zero-Cost Proxies to Estimate Performance of Neural Networks

TL;DR

GreenFactory introduces an ensemble of zero-cost proxies using a random forest to accurately predict neural network performance during architecture search, significantly reducing the need for resource-intensive training.

Contribution

It presents a novel ensemble method that combines multiple zero-cost proxies with a regressor to directly estimate model accuracy, improving generalization across datasets.

Findings

Achieves high Kendall correlations on NATS-Bench datasets.

Demonstrates robustness across multiple datasets and search spaces.

Reduces reliance on training for performance estimation.

Abstract

Determining the performance of a Deep Neural Network during Neural Architecture Search processes is essential for identifying optimal architectures and hyperparameters. Traditionally, this process requires training and evaluation of each network, which is time-consuming and resource-intensive. Zero-cost proxies estimate performance without training, serving as an alternative to traditional training. However, recent proxies often lack generalization across diverse scenarios and provide only relative rankings rather than predicted accuracies. To address these limitations, we propose GreenFactory, an ensemble of zero-cost proxies that leverages a random forest regressor to combine multiple predictors' strengths and directly predict model test accuracy. We evaluate GreenFactory on NATS-Bench, achieving robust results across multiple datasets. Specifically, GreenFactory achieves high Kendall correlations on NATS-Bench-SSS, indicating substantial agreement between its predicted scores and actual performance: 0.907 for CIFAR-10, 0.945 for CIFAR-100, and 0.920 for ImageNet-16-120. Similarly, on NATS-Bench-TSS, we achieve correlations of 0.921 for CIFAR-10, 0.929 for CIFAR-100, and 0.908 for ImageNet-16-120, showcasing its reliability in both search spaces.