Accelerating Reinforcement Learning with Suboptimal Guidance

TL;DR

This paper introduces an adaptive guidance method for reinforcement learning that leverages a Q-filter to incorporate suboptimal controllers, improving exploration efficiency and overall performance in sparse reward environments.

Contribution

It identifies shortcomings in existing Q-filter implementations and proposes modifications that enhance adaptivity and performance in robotic reinforcement learning tasks.

Findings

Modified Q-filter improves guidance adaptivity

Enhanced exploration accelerates learning in sparse rewards

Performance increases across tested robotic environments

Abstract

Reinforcement Learning in domains with sparse rewards is a difficult problem, and a large part of the training process is often spent searching the state space in a more or less random fashion for any learning signals. For control problems, we often have some controller readily available which might be suboptimal but nevertheless solves the problem to some degree. This controller can be used to guide the initial exploration phase of the learning controller towards reward yielding states, reducing the time before refinement of a viable policy can be initiated. In our work, the agent is guided through an auxiliary behaviour cloning loss which is made conditional on a Q-filter, i.e. it is only applied in situations where the critic deems the guiding controller to be better than the agent. The Q-filter provides a natural way to adjust the guidance throughout the training process, allowing the agent to exceed the guiding controller in a manner that is adaptive to the task at hand and the proficiency of the guiding controller. The contribution of this paper lies in identifying shortcomings in previously proposed implementations of the Q-filter concept, and in suggesting some ways these issues can be mitigated. These modifications are tested on the OpenAI Gym Fetch environments, showing clear improvements in adaptivity and yielding increased performance in all robotic environments tested.