Training-free Neural Architecture Search for RNNs and Transformers

TL;DR

This paper introduces a training-free metric called hidden covariance for RNNs, demonstrating its effectiveness in neural architecture search, and highlights the importance of jointly optimizing search space and metrics for transformers.

Contribution

The paper develops a novel training-free metric for RNNs and provides insights into optimizing transformer search spaces for neural architecture search.

Findings

Hidden covariance outperforms existing metrics in predicting RNN performance.

Simplifying the transformer search space improves NAS efficiency.

Joint development of search space and metrics is crucial for effective NAS.

Abstract

Neural architecture search (NAS) has allowed for the automatic creation of new and effective neural network architectures, offering an alternative to the laborious process of manually designing complex architectures. However, traditional NAS algorithms are slow and require immense amounts of computing power. Recent research has investigated training-free NAS metrics for image classification architectures, drastically speeding up search algorithms. In this paper, we investigate training-free NAS metrics for recurrent neural network (RNN) and BERT-based transformer architectures, targeted towards language modeling tasks. First, we develop a new training-free metric, named hidden covariance, that predicts the trained performance of an RNN architecture and significantly outperforms existing training-free metrics. We experimentally evaluate the effectiveness of the hidden covariance metric on the NAS-Bench-NLP benchmark. Second, we find that the current search space paradigm for transformer architectures is not optimized for training-free neural architecture search. Instead, a simple qualitative analysis can effectively shrink the search space to the best performing architectures. This conclusion is based on our investigation of existing training-free metrics and new metrics developed from recent transformer pruning literature, evaluated on our own benchmark of trained BERT architectures. Ultimately, our analysis shows that the architecture search space and the training-free metric must be developed together in order to achieve effective results.