Continuous control with deep reinforcement learning

TL;DR

This paper introduces a deterministic policy gradient-based actor-critic algorithm for continuous control in reinforcement learning, demonstrating its effectiveness across diverse simulated physics tasks and raw pixel inputs.

Contribution

It adapts deep Q-learning ideas to continuous actions, presenting a new model-free algorithm that performs well on complex physics simulations and raw pixel data.

Findings

Successfully solves over 20 physics tasks

Performs comparably to planning algorithms with full domain knowledge

Learns end-to-end policies from raw pixel inputs

Abstract

We adapt the ideas underlying the success of Deep Q-Learning to the continuous action domain. We present an actor-critic, model-free algorithm based on the deterministic policy gradient that can operate over continuous action spaces. Using the same learning algorithm, network architecture and hyper-parameters, our algorithm robustly solves more than 20 simulated physics tasks, including classic problems such as cartpole swing-up, dexterous manipulation, legged locomotion and car driving. Our algorithm is able to find policies whose performance is competitive with those found by a planning algorithm with full access to the dynamics of the domain and its derivatives. We further demonstrate that for many of the tasks the algorithm can learn policies end-to-end: directly from raw pixel inputs.