From Memorization to Creativity: LLM as a Designer of Novel Neural Architectures
Waleed Khalid, Dmitry Ignatov, Radu Timofte

TL;DR
This paper introduces a closed-loop pipeline where a code-oriented LLM evolves neural architectures through iterative synthesis, validation, and filtering, leading to diverse, high-performing models without manual search.
Contribution
It presents a novel iterative self-supervised fine-tuning approach that internalizes architectural priors, enhancing neural architecture generation reliability and diversity.
Findings
Valid generation rate stabilizes at 50.6% on CIFAR-10
Mean accuracy rises from 28.1% to 51.0% across cycles
455 new architectures emerge, not in the original corpus
Abstract
Large language models (LLMs) excel in program synthesis, yet their capacity for neural architecture design -- balancing syntactic reliability, performance, and structural novelty -- remains underexplored. We present a closed-loop architecture synthesis pipeline within the NNGPT framework, in which a code-oriented LLM evolves over 22 supervised fine-tuning cycles. At each cycle, the LLM synthesizes PyTorch convolutional networks, validated via low-fidelity performance signals and filtered via a MinHash--Jaccard criterion to prevent structural redundancy before being incorporated into the LEMUR dataset. High-performing candidates with novel architectures are converted into prompt--code pairs for parameter-efficient LoRA fine-tuning. This feedback loop drives a measurable distributional shift, progressively internalizing empirical architectural priors such that valid and high-performing…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
