Human-Level Control without Server-Grade Hardware

TL;DR

This paper presents a more accessible DQN implementation that significantly reduces training time on standard hardware, making deep reinforcement learning research more feasible without expensive resources.

Contribution

Introduces a concurrent and synchronized execution framework for DQN that efficiently utilizes desktop CPU-GPU systems, reducing training time on standard hardware.

Findings

Training time reduced from 25 hours to 9 hours on a GTX 1080.

Framework is compatible with off-policy deep reinforcement learning methods.

Enables researchers with limited hardware to perform deep RL experiments.

Abstract

Deep Q-Network (DQN) marked a major milestone for reinforcement learning, demonstrating for the first time that human-level control policies could be learned directly from raw visual inputs via reward maximization. Even years after its introduction, DQN remains highly relevant to the research community since many of its innovations have been adopted by successor methods. Nevertheless, despite significant hardware advances in the interim, DQN's original Atari 2600 experiments remain costly to replicate in full. This poses an immense barrier to researchers who cannot afford state-of-the-art hardware or lack access to large-scale cloud computing resources. To facilitate improved access to deep reinforcement learning research, we introduce a DQN implementation that leverages a novel concurrent and synchronized execution framework designed to maximally utilize a heterogeneous CPU-GPU desktop system. With just one NVIDIA GeForce GTX 1080 GPU, our implementation reduces the training time of a 200-million-frame Atari experiment from 25 hours to just 9 hours. The ideas introduced in our paper should be generalizable to a large number of off-policy deep reinforcement learning methods.