Anti-Concentrated Confidence Bonuses for Scalable Exploration

TL;DR

This paper introduces anti-concentrated confidence bonuses for scalable exploration in high-dimensional reinforcement learning, providing efficient approximations of elliptical bonuses and demonstrating competitive performance on Atari benchmarks.

Contribution

It proposes a novel approximation method for elliptical bonuses using ensemble regressors, enabling scalable exploration in high-dimensional settings.

Findings

Achieves $ ilde O(d \, \sqrt{T})$ regret bounds for linear bandits.

Develops a practical deep RL variant competitive with existing heuristics.

Demonstrates effectiveness on Atari benchmarks.

Abstract

Intrinsic rewards play a central role in handling the exploration-exploitation trade-off when designing sequential decision-making algorithms, in both foundational theory and state-of-the-art deep reinforcement learning. The LinUCB algorithm, a centerpiece of the stochastic linear bandits literature, prescribes an elliptical bonus which addresses the challenge of leveraging shared information in large action spaces. This bonus scheme cannot be directly transferred to high-dimensional exploration problems, however, due to the computational cost of maintaining the inverse covariance matrix of action features. We introduce \emph{anti-concentrated confidence bounds} for efficiently approximating the elliptical bonus, using an ensemble of regressors trained to predict random noise from policy network-derived features. Using this approximation, we obtain stochastic linear bandit algorithms which obtain $\tilde O(d \sqrt{T})$ regret bounds for $\mathrm{poly}(d)$ fixed actions. We develop a practical variant for deep reinforcement learning that is competitive with contemporary intrinsic reward heuristics on Atari benchmarks.