Deep Reinforcement Learning for Dynamic Algorithm Configuration: A Case Study on Optimizing OneMax with the (1+($\lambda$,$\lambda$))-GA

TL;DR

This study evaluates deep reinforcement learning algorithms for dynamic algorithm configuration on a simple optimization problem, identifying key challenges and proposing solutions to improve learning stability and exploration.

Contribution

It introduces an adaptive reward shifting mechanism for DDQN, demonstrating improved sample efficiency and stability in controlling the $(1+(mbda,mbda))$-GA on OneMax.

Findings

DDQN with adaptive reward shifting matches theoretical policies' performance

PPO faces variance issues and requires hyperparameter tuning for stability

Standard deep-RL approaches struggle with scalability and exploration in DAC

Abstract

Dynamic Algorithm Configuration (DAC) studies the efficient identification of control policies for parameterized optimization algorithms. Numerous studies leverage Reinforcement Learning (RL) to address DAC challenges; however, applying RL often requires extensive domain expertise. In this work, we conduct a comprehensive study of two deep-RL algorithms--Double Deep Q-Networks (DDQN) and Proximal Policy Optimization (PPO)--for controlling the population size of the $(1+(\lambda,\lambda))$-GA on OneMax instances. Although OneMax is structurally simple, learning effective control policies for the $(1+(\lambda,\lambda))$-GA induces a highly challenging DAC landscape, making it a controlled yet demanding benchmark. Our investigation reveals two fundamental challenges limiting DDQN and PPO: scalability degradation and learning instability, traced to under-exploration and planning horizon coverage. To address under-exploration, we introduce an adaptive reward shifting mechanism that leverages reward distribution statistics to enhance DDQN exploration. This eliminates instance-specific hyperparameter tuning and ensures consistent effectiveness across problem scales. To resolve planning horizon coverage, we demonstrate that undiscounted learning succeeds in DDQN, while PPO faces fundamental variance issues necessitating alternative designs. We further show that while hyperparameter optimization enhances PPO's stability, it consistently fails to identify effective policies. Finally, DDQN with adaptive reward shifting achieves performance comparable to theoretically derived policies with vastly improved sample efficiency, outperforming prior DAC approaches by orders of magnitude. Our findings provide insights into the fundamental obstacles faced by standard deep-RL approaches in this challenging DAC setting and highlight the key methodological ingredients required for effective learning.