Instance Selection for Dynamic Algorithm Configuration with Reinforcement Learning: Improving Generalization

TL;DR

This paper proposes a method to improve the generalization of reinforcement learning agents in dynamic algorithm configuration by selecting representative training instances, leading to more effective and adaptable hyperparameter tuning across diverse problem instances.

Contribution

It introduces an instance selection technique based on meta-features that account for the agent's dynamic behavior, enhancing generalization in DAC tasks.

Findings

Instance selection improves DAC policy performance.

Meta-features based on action and reward trajectories are effective.

Selected training sets outperform full datasets in generalization.

Abstract

Dynamic Algorithm Configuration (DAC) addresses the challenge of dynamically setting hyperparameters of an algorithm for a diverse set of instances rather than focusing solely on individual tasks. Agents trained with Deep Reinforcement Learning (RL) offer a pathway to solve such settings. However, the limited generalization performance of these agents has significantly hindered the application in DAC. Our hypothesis is that a potential bias in the training instances limits generalization capabilities. We take a step towards mitigating this by selecting a representative subset of training instances to overcome overrepresentation and then retraining the agent on this subset to improve its generalization performance. For constructing the meta-features for the subset selection, we particularly account for the dynamic nature of the RL agent by computing time series features on trajectories of actions and rewards generated by the agent's interaction with the environment. Through empirical evaluations on the Sigmoid and CMA-ES benchmarks from the standard benchmark library for DAC, called DACBench, we discuss the potentials of our selection technique compared to training on the entire instance set. Our results highlight the efficacy of instance selection in refining DAC policies for diverse instance spaces.