TART: Token-based Architecture Transformer for Neural Network Performance Prediction

TL;DR

This paper introduces TART, a Transformer-based model that predicts neural network performance without training, facilitating automated neural architecture design and outperforming previous methods on a large benchmark dataset.

Contribution

The paper presents TART, a novel Transformer architecture that predicts neural network performance directly, reducing reliance on manual search and training in neural architecture design.

Findings

TART achieves state-of-the-art performance prediction accuracy.

TART can predict performance without training candidate networks.

The method demonstrates potential for automating neural architecture discovery.

Abstract

In the realm of neural architecture design, achieving high performance is largely reliant on the manual expertise of researchers. Despite the emergence of Neural Architecture Search (NAS) as a promising technique for automating this process, current NAS methods still require human input to expand the search space and cannot generate new architectures. This paper explores the potential of Transformers in comprehending neural architectures and their performance, with the objective of establishing the foundation for utilizing Transformers to generate novel networks. We propose the Token-based Architecture Transformer (TART), which predicts neural network performance without the need to train candidate networks. TART attains state-of-the-art performance on the DeepNets-1M dataset for performance prediction tasks without edge information, indicating the potential of Transformers to aid in discovering novel and high-performing neural architectures.