Improved Exploration in Factored Average-Reward MDPs

TL;DR

This paper introduces DBN-UCRL, a new algorithm for regret minimization in factored average-reward MDPs, which leverages the known factorization to achieve improved theoretical regret bounds and empirical performance.

Contribution

The paper presents DBN-UCRL, a novel regret minimization strategy for factored MDPs that improves upon existing bounds by exploiting the known factorization structure.

Findings

DBN-UCRL achieves lower regret bounds than previous algorithms.

The regret of DBN-UCRL scales favorably with the size of the factored state space.

Numerical experiments show substantial empirical improvement over existing methods.

Abstract

We consider a regret minimization task under the average-reward criterion in an unknown Factored Markov Decision Process (FMDP). More specifically, we consider an FMDP where the state-action space $\mathcal X$ and the state-space $\mathcal S$ admit the respective factored forms of $\mathcal X = \otimes_{i=1}^n \mathcal X_i$ and $\mathcal S=\otimes_{i=1}^m \mathcal S_i$, and the transition and reward functions are factored over $\mathcal X$ and $\mathcal S$. Assuming known factorization structure, we introduce a novel regret minimization strategy inspired by the popular UCRL2 strategy, called DBN-UCRL, which relies on Bernstein-type confidence sets defined for individual elements of the transition function. We show that for a generic factorization structure, DBN-UCRL achieves a regret bound, whose leading term strictly improves over existing regret bounds in terms of the dependencies on the size of $\mathcal S_i$'s and the involved diameter-related terms. We further show that when the factorization structure corresponds to the Cartesian product of some base MDPs, the regret of DBN-UCRL is upper bounded by the sum of regret of the base MDPs. We demonstrate, through numerical experiments on standard environments, that DBN-UCRL enjoys substantially improved regret empirically over existing algorithms that have frequentist regret guarantees.