Playing hard exploration games by watching YouTube

TL;DR

This paper introduces a novel reinforcement learning approach that uses unaligned YouTube videos to guide exploration in sparse reward environments, enabling agents to outperform humans on challenging games without environment rewards.

Contribution

The method leverages self-supervised learning on unaligned videos to create a reward function from YouTube footage, bypassing the need for environment-specific demonstrations.

Findings

Agents surpass human performance on Montezuma's Revenge, Pitfall!, and Private Eye.

The approach works without access to environment rewards or aligned demonstrations.

It demonstrates effective one-shot imitation from unaligned, noisy videos.

Abstract

Deep reinforcement learning methods traditionally struggle with tasks where environment rewards are particularly sparse. One successful method of guiding exploration in these domains is to imitate trajectories provided by a human demonstrator. However, these demonstrations are typically collected under artificial conditions, i.e. with access to the agent's exact environment setup and the demonstrator's action and reward trajectories. Here we propose a two-stage method that overcomes these limitations by relying on noisy, unaligned footage without access to such data. First, we learn to map unaligned videos from multiple sources to a common representation using self-supervised objectives constructed over both time and modality (i.e. vision and sound). Second, we embed a single YouTube video in this representation to construct a reward function that encourages an agent to imitate human gameplay. This method of one-shot imitation allows our agent to convincingly exceed human-level performance on the infamously hard exploration games Montezuma's Revenge, Pitfall! and Private Eye for the first time, even if the agent is not presented with any environment rewards.