An Integrated Approach to Neural Architecture Search for Deep Q-Networks

TL;DR

This paper introduces NAS-DQN, an adaptive neural architecture search method integrated into deep reinforcement learning, which dynamically optimizes network design during training to improve performance and efficiency.

Contribution

It presents NAS-DQN, a novel approach that incorporates online neural architecture search into DRL, enabling dynamic reconfiguration and outperforming fixed architectures.

Findings

NAS-DQN achieves higher final performance

It improves sample efficiency and policy stability

The learned search strategy outperforms random exploration

Abstract

The performance of deep reinforcement learning agents is fundamentally constrained by their neural network architecture, a choice traditionally made through expensive hyperparameter searches and then fixed throughout training. This work investigates whether online, adaptive architecture optimization can escape this constraint and outperform static designs. We introduce NAS-DQN, an agent that integrates a learned neural architecture search controller directly into the DRL training loop, enabling dynamic network reconfiguration based on cumulative performance feedback. We evaluate NAS-DQN against three fixed-architecture baselines and a random search control on a continuous control task, conducting experiments over multiple random seeds. Our results demonstrate that NAS-DQN achieves superior final performance, sample efficiency, and policy stability while incurring negligible computational overhead. Critically, the learned search strategy substantially outperforms both undirected random architecture exploration and poorly-chosen fixed designs, indicating that intelligent, performance-guided search is the key mechanism driving success. These findings establish that architecture adaptation is not merely beneficial but necessary for optimal sample efficiency in online deep reinforcement learning, and suggest that the design of RL agents need not be a static offline choice but can instead be seamlessly integrated as a dynamic component of the learning process itself.