Construction of Hierarchical Neural Architecture Search Spaces based on Context-free Grammars

TL;DR

This paper introduces a unifying framework for designing hierarchical neural architecture search spaces using context-free grammars, enabling exploration of vastly larger and more expressive spaces than previous methods, and proposes an efficient Bayesian Optimization strategy for effective search.

Contribution

The paper presents a novel framework based on context-free grammars for constructing hierarchical NAS spaces and an efficient Bayesian Optimization method to search these large spaces.

Findings

The proposed search space is hundreds of orders of magnitude larger than existing ones.

The Bayesian Optimization strategy outperforms existing NAS methods.

The framework demonstrates versatility across different neural architecture tasks.

Abstract

The discovery of neural architectures from simple building blocks is a long-standing goal of Neural Architecture Search (NAS). Hierarchical search spaces are a promising step towards this goal but lack a unifying search space design framework and typically only search over some limited aspect of architectures. In this work, we introduce a unifying search space design framework based on context-free grammars that can naturally and compactly generate expressive hierarchical search spaces that are 100s of orders of magnitude larger than common spaces from the literature. By enhancing and using their properties, we effectively enable search over the complete architecture and can foster regularity. Further, we propose an efficient hierarchical kernel design for a Bayesian Optimization search strategy to efficiently search over such huge spaces. We demonstrate the versatility of our search space design framework and show that our search strategy can be superior to existing NAS approaches. Code is available at https://github.com/automl/hierarchical_nas_construction.