Distributed Deep Q-Learning

TL;DR

This paper introduces a scalable distributed deep Q-learning framework that enables reinforcement learning agents to learn control policies directly from high-dimensional sensory inputs like raw pixels, demonstrating success on simple games.

Contribution

It adapts the DistBelief framework for distributed reinforcement learning, enabling asynchronous training of deep Q-networks from raw sensory data.

Findings

Successfully learned control policies from raw pixels

Achieved reasonable performance on simple game tasks

Demonstrated scalability with multiple machines

Abstract

We propose a distributed deep learning model to successfully learn control policies directly from high-dimensional sensory input using reinforcement learning. The model is based on the deep Q-network, a convolutional neural network trained with a variant of Q-learning. Its input is raw pixels and its output is a value function estimating future rewards from taking an action given a system state. To distribute the deep Q-network training, we adapt the DistBelief software framework to the context of efficiently training reinforcement learning agents. As a result, the method is completely asynchronous and scales well with the number of machines. We demonstrate that the deep Q-network agent, receiving only the pixels and the game score as inputs, was able to achieve reasonable success on a simple game with minimal parameter tuning.