Sample Efficient Robot Learning with Structured World Models

TL;DR

This paper investigates how structured world models, especially using keypoints, enhance sample efficiency and performance in robotic cloth-folding tasks, demonstrating significant improvements over unstructured observations.

Contribution

It introduces the use of structured feature spaces like keypoints in world models for robotic manipulation, showing improved efficiency and performance in cloth-folding tasks.

Findings

Keypoints increased task performance by 50%.

Structured feature spaces improved sample efficiency.

State transition predictor had no notable effect.

Abstract

Reinforcement learning has been demonstrated as a flexible and effective approach for learning a range of continuous control tasks, such as those used by robots to manipulate objects in their environment. But in robotics particularly, real-world rollouts are costly, and sample efficiency can be a major limiting factor when learning a new skill. In game environments, the use of world models has been shown to improve sample efficiency while still achieving good performance, especially when images or other rich observations are provided. In this project, we explore the use of a world model in a deformable robotic manipulation task, evaluating its effect on sample efficiency when learning to fold a cloth in simulation. We compare the use of RGB image observation with a feature space leveraging built-in structure (keypoints representing the cloth configuration), a common approach in robot skill learning, and compare the impact on task performance and learning efficiency with and without the world model. Our experiments showed that the usage of keypoints increased the performance of the best model on the task by 50%, and in general, the use of a learned or constructed reduced feature space improved task performance and sample efficiency. The use of a state transition predictor(MDN-RNN) in our world models did not have a notable effect on task performance.