Hyperparameter Tuning for Deep Reinforcement Learning Applications

TL;DR

This paper introduces a distributed genetic algorithm framework to efficiently tune hyperparameters in deep reinforcement learning, enhancing training speed, robustness, and deployment readiness across diverse applications.

Contribution

It presents a novel scalable genetic algorithm approach for hyperparameter tuning in deep RL, outperforming Bayesian methods in efficiency and robustness.

Findings

Fewer training episodes needed with more generations

Improved robustness of RL models for deployment

Scalable tuning across simple to complex RL problems

Abstract

Reinforcement learning (RL) applications, where an agent can simply learn optimal behaviors by interacting with the environment, are quickly gaining tremendous success in a wide variety of applications from controlling simple pendulums to complex data centers. However, setting the right hyperparameters can have a huge impact on the deployed solution performance and reliability in the inference models, produced via RL, used for decision-making. Hyperparameter search itself is a laborious process that requires many iterations and computationally expensive to find the best settings that produce the best neural network architectures. In comparison to other neural network architectures, deep RL has not witnessed much hyperparameter tuning, due to its algorithm complexity and simulation platforms needed. In this paper, we propose a distributed variable-length genetic algorithm framework to systematically tune hyperparameters for various RL applications, improving training time and robustness of the architecture, via evolution. We demonstrate the scalability of our approach on many RL problems (from simple gyms to complex applications) and compared with Bayesian approach. Our results show that with more generations, optimal solutions that require fewer training episodes and are computationally cheap while being more robust for deployment. Our results are imperative to advance deep reinforcement learning controllers for real-world problems.