AIO-P: Expanding Neural Performance Predictors Beyond Image Classification

TL;DR

AIO-P is a versatile neural network performance predictor trained across multiple vision tasks and architectures, enabling accurate performance estimation and ranking for unseen architectures and tasks, thus reducing evaluation costs.

Contribution

The paper introduces AIO-P, a novel universal predictor trained on diverse architecture and task data, capable of transfer learning to unseen neural architectures and downstream vision tasks.

Findings

Achieves MAE below 1% and SRCC above 0.5 on various CV tasks.

Effectively transfers to unseen architectures for accurate ranking.

Outperforms existing task-dependent predictors.

Abstract

Evaluating neural network performance is critical to deep neural network design but a costly procedure. Neural predictors provide an efficient solution by treating architectures as samples and learning to estimate their performance on a given task. However, existing predictors are task-dependent, predominantly estimating neural network performance on image classification benchmarks. They are also search-space dependent; each predictor is designed to make predictions for a specific architecture search space with predefined topologies and set of operations. In this paper, we propose a novel All-in-One Predictor (AIO-P), which aims to pretrain neural predictors on architecture examples from multiple, separate computer vision (CV) task domains and multiple architecture spaces, and then transfer to unseen downstream CV tasks or neural architectures. We describe our proposed techniques for general graph representation, efficient predictor pretraining and knowledge infusion techniques, as well as methods to transfer to downstream tasks/spaces. Extensive experimental results show that AIO-P can achieve Mean Absolute Error (MAE) and Spearman's Rank Correlation (SRCC) below 1% and above 0.5, respectively, on a breadth of target downstream CV tasks with or without fine-tuning, outperforming a number of baselines. Moreover, AIO-P can directly transfer to new architectures not seen during training, accurately rank them and serve as an effective performance estimator when paired with an algorithm designed to preserve performance while reducing FLOPs.