Probing Transfer in Deep Reinforcement Learning without Task Engineering

TL;DR

This paper evaluates the use of Atari game curricula as a benchmark for transfer learning in deep reinforcement learning, demonstrating significant effects of game design factors on learning and transfer performance without manual task engineering.

Contribution

It introduces a multi-factorial Atari curriculum benchmark and uses ANOVA to analyze how design factors influence transfer, highlighting the potential for zero-shot transfer and the need for algorithmic improvements.

Findings

Game design factors significantly affect learning and transfer.

Zero-shot transfer is possible between basic and modified games.

Value-function finetuning often yields positive transfer, but room for improvement exists.

Abstract

We evaluate the use of original game curricula supported by the Atari 2600 console as a heterogeneous transfer benchmark for deep reinforcement learning agents. Game designers created curricula using combinations of several discrete modifications to the basic versions of games such as Space Invaders, Breakout and Freeway, making them progressively more challenging for human players. By formally organising these modifications into several factors of variation, we are able to show that Analyses of Variance (ANOVA) are a potent tool for studying the effects of human-relevant domain changes on the learning and transfer performance of a deep reinforcement learning agent. Since no manual task engineering is needed on our part, leveraging the original multi-factorial design avoids the pitfalls of unintentionally biasing the experimental setup. We find that game design factors have a large and statistically significant impact on an agent's ability to learn, and so do their combinatorial interactions. Furthermore, we show that zero-shot transfer from the basic games to their respective variations is possible, but the variance in performance is also largely explained by interactions between factors. As such, we argue that Atari game curricula offer a challenging benchmark for transfer learning in RL, that can help the community better understand the generalisation capabilities of RL agents along dimensions which meaningfully impact human generalisation performance. As a start, we report that value-function finetuning of regularly trained agents achieves positive transfer in a majority of cases, but significant headroom for algorithmic innovation remains. We conclude with the observation that selective transfer from multiple variants could further improve performance.