Distilling Reinforcement Learning Tricks for Video Games

TL;DR

This paper identifies and distills practical engineering tricks used in reinforcement learning for video games, demonstrating how these techniques can significantly improve the performance of standard deep Q-learning agents.

Contribution

It provides a systematic analysis of RL tricks from state-of-the-art results and explores their effectiveness in enhancing deep Q-learning agents.

Findings

RL tricks improve agent performance

Distilled tricks enable better sample efficiency

Framework facilitates combining RL methods with domain tricks

Abstract

Reinforcement learning (RL) research focuses on general solutions that can be applied across different domains. This results in methods that RL practitioners can use in almost any domain. However, recent studies often lack the engineering steps ("tricks") which may be needed to effectively use RL, such as reward shaping, curriculum learning, and splitting a large task into smaller chunks. Such tricks are common, if not necessary, to achieve state-of-the-art results and win RL competitions. To ease the engineering efforts, we distill descriptions of tricks from state-of-the-art results and study how well these tricks can improve a standard deep Q-learning agent. The long-term goal of this work is to enable combining proven RL methods with domain-specific tricks by providing a unified software framework and accompanying insights in multiple domains.