RevoNAD: Reflective Evolutionary Exploration for Neural Architecture Design

TL;DR

RevoNAD introduces a reflective evolutionary framework that combines LLM reasoning with feedback-guided search to design neural architectures more effectively, achieving state-of-the-art results across multiple datasets.

Contribution

It presents a novel multi-round consensus, adaptive exploration, and Pareto-guided selection to improve neural architecture search using LLMs.

Findings

Achieves state-of-the-art performance on CIFAR and ImageNet datasets.

Effectively balances exploration and exploitation in architecture search.

Validates the approach through ablation and transfer studies.

Abstract

Recent progress in leveraging large language models (LLMs) has enabled Neural Architecture Design (NAD) systems to generate new architecture not limited from manually predefined search space. Nevertheless, LLM-driven generation remains challenging: the token-level design loop is discrete and non-differentiable, preventing feedback from smoothly guiding architectural improvement. These methods, in turn, commonly suffer from mode collapse into redundant structures or drift toward infeasible designs when constructive reasoning is not well grounded. We introduce RevoNAD, a reflective evolutionary orchestrator that effectively bridges LLM-based reasoning with feedback-aligned architectural search. First, RevoNAD presents a Multi-round Multi-expert Consensus to transfer isolated design rules into meaningful architectural clues. Then, Adaptive Reflective Exploration adjusts the degree of exploration leveraging reward variance; it explores when feedback is uncertain and refines when stability is reached. Finally, Pareto-guided Evolutionary Selection effectively promotes architectures that jointly optimize accuracy, efficiency, latency, confidence, and structural diversity. Across CIFAR10, CIFAR100, ImageNet16-120, COCO-5K, and Cityscape, RevoNAD achieves state-of-the-art performance. Ablation and transfer studies further validate the effectiveness of RevoNAD in allowing practically reliable, and deployable neural architecture design.