Diagnosing and exploiting the computational demands of videos games for deep reinforcement learning

TL;DR

This paper introduces the Learning Challenge Diagnosticator (LCD), a tool that measures perceptual and reinforcement learning demands in video games, helping to understand and improve deep reinforcement learning algorithms.

Contribution

The paper presents the LCD tool and a new taxonomy of challenges in video game benchmarks, aiding in diagnosing and enhancing deep reinforcement learning performance.

Findings

LCD reliably measures perceptual and reinforcement demands.

Taxonomy of challenges guides algorithmic improvements.

Identifies failure cases in current deep RL benchmarks.

Abstract

Humans learn by interacting with their environments and perceiving the outcomes of their actions. A landmark in artificial intelligence has been the development of deep reinforcement learning (dRL) algorithms capable of doing the same in video games, on par with or better than humans. However, it remains unclear whether the successes of dRL models reflect advances in visual representation learning, the effectiveness of reinforcement learning algorithms at discovering better policies, or both. To address this question, we introduce the Learning Challenge Diagnosticator (LCD), a tool that separately measures the perceptual and reinforcement learning demands of a task. We use LCD to discover a novel taxonomy of challenges in the Procgen benchmark, and demonstrate that these predictions are both highly reliable and can instruct algorithmic development. More broadly, the LCD reveals multiple failure cases that can occur when optimizing dRL algorithms over entire video game benchmarks like Procgen, and provides a pathway towards more efficient progress.