SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning

TL;DR

This paper introduces SOLAR, a deep structured representation learning method that enhances model-based reinforcement learning with image observations, enabling efficient control and superior performance in robotics tasks.

Contribution

The paper proposes a novel representation learning approach tailored for model-based RL with complex observations, facilitating the use of LQR in image-based control tasks.

Findings

Outperforms existing model-based RL methods in final performance

Achieves higher data efficiency compared to model-free RL

Successfully applied to real-world robotic manipulation from images

Abstract

Model-based reinforcement learning (RL) has proven to be a data efficient approach for learning control tasks but is difficult to utilize in domains with complex observations such as images. In this paper, we present a method for learning representations that are suitable for iterative model-based policy improvement, even when the underlying dynamical system has complex dynamics and image observations, in that these representations are optimized for inferring simple dynamics and cost models given data from the current policy. This enables a model-based RL method based on the linear-quadratic regulator (LQR) to be used for systems with image observations. We evaluate our approach on a range of robotics tasks, including manipulation with a real-world robotic arm directly from images. We find that our method produces substantially better final performance than other model-based RL methods while being significantly more efficient than model-free RL.