Guided Policy Search with Delayed Sensor Measurements

TL;DR

This paper extends guided policy search to non-markovian systems with delayed sensor measurements by using simple state augmentation, enabling effective reinforcement learning in such challenging environments.

Contribution

It introduces a method to adapt guided policy search for non-markovian dynamics caused by sensor delays through state augmentation.

Findings

Effective policy learned for liquid pouring task with delayed sensors

Simple state augmentation suffices for non-markovian dynamics

Guided policy search successfully applied beyond Markovian assumptions

Abstract

Guided policy search is a method for reinforcement learning that trains a general policy for accomplishing a given task by guiding the learning of the policy with multiple guiding distributions. Guided policy search relies on learning an underlying dynamical model of the environment and then, at each iteration of the algorithm, using that model to gradually improve the policy. This model, though, often makes the assumption that the environment dynamics are markovian, e.g., depend only on the current state and control signal. In this paper we apply guided policy search to a problem with non-markovian dynamics. Specifically, we apply it to the problem of pouring a precise amount of liquid from a cup into a bowl, where many of the sensor measurements experience non-trivial amounts of delay. We show that, with relatively simple state augmentation, guided policy search can be extended to non-markovian dynamical systems, where the non-markovianess is caused by delayed sensor readings.