Deep Exploration via Bootstrapped DQN

TL;DR

Bootstrapped DQN introduces a randomized value function approach for deep, efficient exploration in reinforcement learning, significantly improving learning speed and performance in complex environments like Atari games.

Contribution

It presents a simple, effective algorithm for deep exploration using randomized value functions, outperforming traditional methods like epsilon-greedy in complex tasks.

Findings

Faster learning in stochastic MDPs.

Improved performance across Atari games.

Exponential speedup in exploration efficiency.

Abstract

Efficient exploration in complex environments remains a major challenge for reinforcement learning. We propose bootstrapped DQN, a simple algorithm that explores in a computationally and statistically efficient manner through use of randomized value functions. Unlike dithering strategies such as epsilon-greedy exploration, bootstrapped DQN carries out temporally-extended (or deep) exploration; this can lead to exponentially faster learning. We demonstrate these benefits in complex stochastic MDPs and in the large-scale Arcade Learning Environment. Bootstrapped DQN substantially improves learning times and performance across most Atari games.