LLM-Driven Composite Neural Architecture Search for Multi-Source RL State Encoding

TL;DR

This paper introduces an LLM-guided neural architecture search method for designing multi-source state encoders in reinforcement learning, improving sample efficiency and performance over traditional methods.

Contribution

It presents a novel LLM-driven NAS pipeline that leverages language priors and intermediate signals to optimize composite neural architectures for multi-source RL tasks.

Findings

Achieves higher-performing architectures with fewer evaluations.

Outperforms traditional NAS baselines and GENIUS framework.

Demonstrates effectiveness on a mixed-autonomy traffic control task.

Abstract

Designing state encoders for reinforcement learning (RL) with multiple information sources -- such as sensor measurements, time-series signals, image observations, and textual instructions -- remains underexplored and often requires manual design. We formalize this challenge as a problem of composite neural architecture search (NAS), where multiple source-specific modules and a fusion module are jointly optimized. Existing NAS methods overlook useful side information from the intermediate outputs of these modules -- such as their representation quality -- limiting sample efficiency in multi-source RL settings. To address this, we propose an LLM-driven NAS pipeline in which the LLM serves as a neural architecture design agent, leveraging language-model priors and intermediate-output signals to guide sample-efficient search for high-performing composite state encoders. On a mixed-autonomy traffic control task, our approach discovers higher-performing architectures with fewer candidate evaluations than traditional NAS baselines and the LLM-based GENIUS framework.