LLM as a Tool, Not an Agent: Code-Mined Tree Transformations for Neural Architecture Search

TL;DR

This paper introduces LLMasTool, a hierarchical tree-based NAS framework that combines reliable tree transformations with LLM assistance to enable open-ended neural architecture search beyond LLM biases.

Contribution

The paper presents a novel NAS approach that extracts reusable modules from source code, representing architectures as hierarchical trees for stable evolution guided by Bayesian modeling.

Findings

Improves NAS performance on CIFAR-10, CIFAR-100, and ImageNet16-120 datasets.

Uses a diversity-guided algorithm for efficient exploration.

Ensures executable architectures through tree transformations rather than code generation.

Abstract

Neural Architecture Search (NAS) aims to automatically discover high-performing deep neural network (DNN) architectures. However, conventional algorithm-driven NAS relies on carefully hand-crafted search spaces to ensure executability, which restricts open-ended exploration. Recent coding-based agentic approaches using large language models (LLMs) reduce manual design, but current LLMs struggle to reliably generate complex, valid architectures, and their proposals are often biased toward a narrow set of patterns observed in their training data. To bridge reliable algorithmic search with powerful LLM assistance, we propose LLMasTool, a hierarchical tree-based NAS framework for stable and open-ended model evolution. Our method automatically extracts reusable modules from arbitrary source code and represents full architectures as hierarchical trees, enabling evolution through reliable tree transformations rather than code generation. At each evolution step, coarse-level planning is governed by a diversity-guided algorithm that leverages Bayesian modeling to improve exploration efficiency, while the LLM resolves the remaining degrees of freedom to ensure a meaningful evolutionary trajectory and an executable generated architecture. With this formulation, instead of fully agentic LLM approaches, our method explores diverse directions beyond the inherent biases in the LLM. Our method improves over existing NAS methods by 0.69, 1.83, and 2.68 points on CIFAR-10, CIFAR-100, and ImageNet16-120, demonstrating its effectiveness.