SEKI: Self-Evolution and Knowledge Inspiration based Neural Architecture Search via Large Language Models

TL;DR

SEKI is a novel LLM-based neural architecture search method that combines self-evolution and knowledge distillation to efficiently discover high-performance neural architectures without domain-specific data.

Contribution

It introduces a two-stage NAS framework leveraging LLMs, achieving state-of-the-art results with minimal computational resources and no domain-specific data.

Findings

Achieves SOTA performance across multiple datasets and search spaces.

Requires only 0.05 GPU-days for architecture search.

Demonstrates strong generalization across various tasks.

Abstract

We introduce SEKI, a novel large language model (LLM)-based neural architecture search (NAS) method. Inspired by the chain-of-thought (CoT) paradigm in modern LLMs, SEKI operates in two key stages: self-evolution and knowledge distillation. In the self-evolution stage, LLMs initially lack sufficient reference examples, so we implement an iterative refinement mechanism that enhances architectures based on performance feedback. Over time, this process accumulates a repository of high-performance architectures. In the knowledge distillation stage, LLMs analyze common patterns among these architectures to generate new, optimized designs. Combining these two stages, SEKI greatly leverages the capacity of LLMs on NAS and without requiring any domain-specific data. Experimental results show that SEKI achieves state-of-the-art (SOTA) performance across various datasets and search spaces while requiring only 0.05 GPU-days, outperforming existing methods in both efficiency and accuracy. Furthermore, SEKI demonstrates strong generalization capabilities, achieving SOTA-competitive results across multiple tasks.