NADER: Neural Architecture Design via Multi-Agent Collaboration

TL;DR

NADER introduces a multi-agent collaboration framework using LLMs and graph-based architecture representations to improve neural architecture design beyond traditional search spaces.

Contribution

It presents a novel multi-agent LLM-based approach with a learning reflector and graph representations for more effective neural architecture search.

Findings

Outperforms state-of-the-art NAS methods on benchmark tasks.

Effectively explores architectures beyond predefined search spaces.

Demonstrates improved efficiency and architecture quality.

Abstract

Designing effective neural architectures poses a significant challenge in deep learning. While Neural Architecture Search (NAS) automates the search for optimal architectures, existing methods are often constrained by predetermined search spaces and may miss critical neural architectures. In this paper, we introduce NADER (Neural Architecture Design via multi-agEnt collaboRation), a novel framework that formulates neural architecture design (NAD) as a LLM-based multi-agent collaboration problem. NADER employs a team of specialized agents to enhance a base architecture through iterative modification. Current LLM-based NAD methods typically operate independently, lacking the ability to learn from past experiences, which results in repeated mistakes and inefficient exploration. To address this issue, we propose the Reflector, which effectively learns from immediate feedback and long-term experiences. Additionally, unlike previous LLM-based methods that use code to represent neural architectures, we utilize a graph-based representation. This approach allows agents to focus on design aspects without being distracted by coding. We demonstrate the effectiveness of NADER in discovering high-performing architectures beyond predetermined search spaces through extensive experiments on benchmark tasks, showcasing its advantages over state-of-the-art methods. The codes will be released soon.