UCB Exploration via Q-Ensembles

TL;DR

This paper introduces a UCB-based exploration strategy using Q-ensembles for deep reinforcement learning, demonstrating significant performance improvements on Atari benchmarks.

Contribution

It adapts UCB exploration from bandit algorithms to deep Q-learning using ensembles, providing a novel approach for more effective exploration.

Findings

Significant gains on Atari benchmark tasks

Effective adaptation of UCB exploration to deep Q-learning

Q-ensembles improve exploration efficiency

Abstract

We show how an ensemble of $Q^*$-functions can be leveraged for more effective exploration in deep reinforcement learning. We build on well established algorithms from the bandit setting, and adapt them to the $Q$-learning setting. We propose an exploration strategy based on upper-confidence bounds (UCB). Our experiments show significant gains on the Atari benchmark.