BLOX: Macro Neural Architecture Search Benchmark and Algorithms

TL;DR

BLOX provides a comprehensive benchmark and analysis of NAS algorithms on macro search spaces, enabling systematic evaluation of their performance and scalability across diverse hardware platforms.

Contribution

This work introduces BLOX, a large-scale benchmark with 91k models for macro NAS, facilitating systematic comparison of NAS algorithms on diverse hardware.

Findings

Macro search space can lead to better model performance.

Existing NAS algorithms show varying effectiveness on macro spaces.

Benchmark enables evaluation of scalability and hardware performance.

Abstract

Neural architecture search (NAS) has been successfully used to design numerous high-performance neural networks. However, NAS is typically compute-intensive, so most existing approaches restrict the search to decide the operations and topological structure of a single block only, then the same block is stacked repeatedly to form an end-to-end model. Although such an approach reduces the size of search space, recent studies show that a macro search space, which allows blocks in a model to be different, can lead to better performance. To provide a systematic study of the performance of NAS algorithms on a macro search space, we release Blox - a benchmark that consists of 91k unique models trained on the CIFAR-100 dataset. The dataset also includes runtime measurements of all the models on a diverse set of hardware platforms. We perform extensive experiments to compare existing algorithms that are well studied on cell-based search spaces, with the emerging blockwise approaches that aim to make NAS scalable to much larger macro search spaces. The benchmark and code are available at https://github.com/SamsungLabs/blox.