Deep Spatial Autoencoders for Visuomotor Learning

TL;DR

This paper introduces a deep spatial autoencoder that automatically learns environment features from camera images, enabling robots to perform visuomotor tasks with reinforcement learning and closed-loop control.

Contribution

It presents a novel method combining deep spatial autoencoders with reinforcement learning for automatic state representation and manipulation in robotic tasks.

Findings

Successfully learned to manipulate objects like blocks, rice bags, and ropes.

Automatically tracked task-relevant objects without manual feature engineering.

Enabled dynamic, closed-loop control for various robotic manipulation tasks.

Abstract

Reinforcement learning provides a powerful and flexible framework for automated acquisition of robotic motion skills. However, applying reinforcement learning requires a sufficiently detailed representation of the state, including the configuration of task-relevant objects. We present an approach that automates state-space construction by learning a state representation directly from camera images. Our method uses a deep spatial autoencoder to acquire a set of feature points that describe the environment for the current task, such as the positions of objects, and then learns a motion skill with these feature points using an efficient reinforcement learning method based on local linear models. The resulting controller reacts continuously to the learned feature points, allowing the robot to dynamically manipulate objects in the world with closed-loop control. We demonstrate our method with a PR2 robot on tasks that include pushing a free-standing toy block, picking up a bag of rice using a spatula, and hanging a loop of rope on a hook at various positions. In each task, our method automatically learns to track task-relevant objects and manipulate their configuration with the robot's arm.