3D Simulation for Robot Arm Control with Deep Q-Learning

TL;DR

This paper demonstrates training a deep Q-learning based controller for a 7-DOF robot arm in 3D simulation, which can transfer to real robots for grasping tasks, reducing reliance on real-world data.

Contribution

It introduces a simulation-based deep Q-learning approach for robot arm control that transfers policies directly to real robots without additional training.

Findings

Successful transfer of policies from simulation to real robot

Structured reward functions improve learning efficiency

Controller operates using only visual input

Abstract

Recent trends in robot arm control have seen a shift towards end-to-end solutions, using deep reinforcement learning to learn a controller directly from raw sensor data, rather than relying on a hand-crafted, modular pipeline. However, the high dimensionality of the state space often means that it is impractical to generate sufficient training data with real-world experiments. As an alternative solution, we propose to learn a robot controller in simulation, with the potential of then transferring this to a real robot. Building upon the recent success of deep Q-networks, we present an approach which uses 3D simulations to train a 7-DOF robotic arm in a control task without any prior knowledge. The controller accepts images of the environment as its only input, and outputs motor actions for the task of locating and grasping a cube, over a range of initial configurations. To encourage efficient learning, a structured reward function is designed with intermediate rewards. We also present preliminary results in direct transfer of policies over to a real robot, without any further training.