Low Dimensional State Representation Learning with Reward-shaped Priors

TL;DR

This paper introduces a method for learning low-dimensional state representations in reinforcement learning for robotics, using reward-shaped priors to improve sample efficiency and facilitate faster policy learning.

Contribution

It proposes an unsupervised learning approach that incorporates environment and task priors into the state representation, enhancing sample efficiency in robotic RL tasks.

Findings

Effective in simulation navigation tasks

Successful transfer to real robot experiments

Improved sample efficiency in policy learning

Abstract

Reinforcement Learning has been able to solve many complicated robotics tasks without any need for feature engineering in an end-to-end fashion. However, learning the optimal policy directly from the sensory inputs, i.e the observations, often requires processing and storage of a huge amount of data. In the context of robotics, the cost of data from real robotics hardware is usually very high, thus solutions that achieve high sample-efficiency are needed. We propose a method that aims at learning a mapping from the observations into a lower-dimensional state space. This mapping is learned with unsupervised learning using loss functions shaped to incorporate prior knowledge of the environment and the task. Using the samples from the state space, the optimal policy is quickly and efficiently learned. We test the method on several mobile robot navigation tasks in a simulation environment and also on a real robot.