Tighter Problem-Dependent Regret Bounds in Reinforcement Learning without Domain Knowledge using Value Function Bounds

TL;DR

This paper introduces a reinforcement learning algorithm with problem-dependent regret bounds that are tighter when the environment has small variance in value functions, without needing prior knowledge of environmental norms.

Contribution

It develops a new RL algorithm for finite horizon MDPs that achieves state-of-the-art worst-case regret bounds and adapts to environment variance without prior bounds.

Findings

Achieves tight worst-case regret bounds in finite horizon MDPs.

Provides substantially tighter bounds for environments with small variance.

Addresses an open question by removing H-dependence in regret bounds.

Abstract

Strong worst-case performance bounds for episodic reinforcement learning exist but fortunately in practice RL algorithms perform much better than such bounds would predict. Algorithms and theory that provide strong problem-dependent bounds could help illuminate the key features of what makes a RL problem hard and reduce the barrier to using RL algorithms in practice. As a step towards this we derive an algorithm for finite horizon discrete MDPs and associated analysis that both yields state-of-the art worst-case regret bounds in the dominant terms and yields substantially tighter bounds if the RL environment has small environmental norm, which is a function of the variance of the next-state value functions. An important benefit of our algorithmic is that it does not require apriori knowledge of a bound on the environmental norm. As a result of our analysis, we also help address an open learning theory question~\cite{jiang2018open} about episodic MDPs with a constant upper-bound on the sum of rewards, providing a regret bound with no $H$-dependence in the leading term that scales a polynomial function of the number of episodes.