Implementing the Deep Q-Network

TL;DR

This paper reproduces and analyzes the implementation of the Deep Q-Network, highlighting key details for replication, discussing performance improvements, and providing a versatile implementation for various domains.

Contribution

It offers detailed insights into replicating DQN results, clarifies previously under-documented implementation details, and presents a flexible, performance-optimized implementation.

Findings

Reproduced DQN results with high fidelity

Identified critical implementation details for replication

Enhanced computational performance of DQN implementations

Abstract

The Deep Q-Network proposed by Mnih et al. [2015] has become a benchmark and building point for much deep reinforcement learning research. However, replicating results for complex systems is often challenging since original scientific publications are not always able to describe in detail every important parameter setting and software engineering solution. In this paper, we present results from our work reproducing the results of the DQN paper. We highlight key areas in the implementation that were not covered in great detail in the original paper to make it easier for researchers to replicate these results, including termination conditions and gradient descent algorithms. Finally, we discuss methods for improving the computational performance and provide our own implementation that is designed to work with a range of domains, and not just the original Arcade Learning Environment [Bellemare et al., 2013].