Structured Progressive Knowledge Activation for LLM-Driven Neural Architecture Search

TL;DR

This paper introduces SPARK, a method that improves neural architecture search by activating relevant priors and reducing functional entanglement, leading to more efficient and reliable architecture modifications using LLMs.

Contribution

The paper proposes Structured Progressive Knowledge Activation (SPARK), a novel approach that explicitly selects and conditions on functional factors to enhance LLM-driven NAS.

Findings

SPARK achieves a 28.1x speedup in architecture evolution.

SPARK improves OOD accuracy by 22.9% relative.

Reduces side effects of local edits in NAS.

Abstract

This paper focuses on a key challenge in Neural Architecture Search (NAS): integrating established architectural knowledge while exploring new designs under expensive evaluations. Large language models (LLMs) are a promising assistant for NAS because they can translate rich architectural and coding priors into executable code edits. However, in practice, seemingly local revisions often propagate into non-local behavioral and performance shifts because a single edit can inadvertently couple multiple interacting functional factors, a phenomenon we refer to as functional entanglement. To make LLM knowledge usable under such entanglement, we propose Structured Progressive Knowledge Activation (SPARK), which activates relevant priors by explicitly selecting the functional factor to modify and conditioning the edit on that factor. This factor-conditioned editing reduces entangled side effects and yields more targeted, reliable architecture modifications. On CLRS-DFS, SPARK achieves a 28.1x sample-efficient architecture evolution speedup and yields a 22.9 percent relative improvement in OOD accuracy.