Reinforcement Learning using Guided Observability

TL;DR

This paper introduces PO-GRL, a simple method that gradually transitions from full to partial observability during training, significantly improving reinforcement learning performance in various partially observable environments.

Contribution

The paper proposes PO-GRL, a novel approach that leverages full state information during training to enhance RL in partially observable settings, applicable across many RL methods.

Findings

PO-GRL improves performance in POMDP benchmarks.

PO-GRL enhances RL in continuous MuJoCo and OpenAI gym tasks.

PO-GRL successfully applied to a real robot in the ball-in-the-cup task.

Abstract

Due to recent breakthroughs, reinforcement learning (RL) has demonstrated impressive performance in challenging sequential decision-making problems. However, an open question is how to make RL cope with partial observability which is prevalent in many real-world problems. Contrary to contemporary RL approaches, which focus mostly on improved memory representations or strong assumptions about the type of partial observability, we propose a simple but efficient approach that can be applied together with a wide variety of RL methods. Our main insight is that smoothly transitioning from full observability to partial observability during the training process yields a high performance policy. The approach, called partially observable guided reinforcement learning (PO-GRL), allows to utilize full state information during policy optimization without compromising the optimality of the final policy. A comprehensive evaluation in discrete partially observableMarkov decision process (POMDP) benchmark problems and continuous partially observable MuJoCo and OpenAI gym tasks shows that PO-GRL improves performance. Finally, we demonstrate PO-GRL in the ball-in-the-cup task on a real Barrett WAM robot under partial observability.