Direct Random Search for Fine Tuning of Deep Reinforcement Learning Policies

TL;DR

This paper demonstrates that direct random search effectively fine-tunes deterministic policies in deep reinforcement learning, resulting in more consistent and higher-performing agents across various environments.

Contribution

It introduces a simple yet effective method for fine-tuning DRL policies through direct random search, improving performance and consistency.

Findings

More consistent agent performance across environments.

Higher average rewards compared to baseline policies.

Effective extension to state space reduction techniques.

Abstract

Researchers have demonstrated that Deep Reinforcement Learning (DRL) is a powerful tool for finding policies that perform well on complex robotic systems. However, these policies are often unpredictable and can induce highly variable behavior when evaluated with only slightly different initial conditions. Training considerations constrain DRL algorithm designs in that most algorithms must use stochastic policies during training. The resulting policy used during deployment, however, can and frequently is a deterministic one that uses the Maximum Likelihood Action (MLA) at each step. In this work, we show that a direct random search is very effective at fine-tuning DRL policies by directly optimizing them using deterministic rollouts. We illustrate this across a large collection of reinforcement learning environments, using a wide variety of policies obtained from different algorithms. Our results show that this method yields more consistent and higher performing agents on the environments we tested. Furthermore, we demonstrate how this method can be used to extend our previous work on shrinking the dimensionality of the reachable state space of closed-loop systems run under Deep Neural Network (DNN) policies.