Structuring Open-Ended NAS: Semi-Automated Design Knowledge Structuring with LLMs for Efficient Neural Architecture Search

TL;DR

This paper introduces a semi-automated approach to structure design knowledge for open-ended neural architecture search using LLMs, improving exploration efficiency and discovering high-performing models.

Contribution

It proposes a method to semi-automatically structure architectural knowledge with LLMs and introduces FairNAD, a mutation strategy for efficient search in large, structured spaces.

Findings

Achieved 0.84, 2.17, and 2.35 points improvements on CIFAR-10, CIFAR-100, and ImageNet16-120.

Demonstrated effective exploration of large search spaces with the proposed mutation strategy.

Enabled high-quality architecture discovery through structured knowledge and LLM-guided search.

Abstract

Current neural architecture search (NAS) methods are often limited by their predefined, restrictive search spaces. While recent large language model (LLM)-assisted NAS methods enable open-ended search spaces, they often suffer from inefficient exploration due to biased or low-quality design ideas. To address these issues, we propose to semi-automatically structure model design knowledge to guide the search process. Our approach first defines a high-level structural template of architectural attributes. An LLM then populates this template by analyzing papers, creating a rich and diverse search space that embodies this structured design knowledge. To efficiently explore this vast space, we introduce FairNAD, using a multi-type mutation that enables broad exploration through mutation with fair idea sampling, Pareto-aware mutation, LLM-driven iterative mutation, and a fine-grained feedback loop. We demonstrate the effectiveness of FairNAD in discovering high-performing architectures that yield 0.84, 2.17, and 2.35 points improvement on CIFAR-10, CIFAR-100, and ImageNet16-120, respectively, compared to current state-of-the-art methods.