Online Reinforcement Learning for Periodic MDP

TL;DR

This paper introduces PUCRL2, an algorithm for learning in periodic MDPs where transition and reward functions change periodically, achieving sub-linear regret with respect to horizon length.

Contribution

The paper formulates periodic MDPs as stationary by augmenting state space and proposes PUCRL2, a novel algorithm with regret bounds that depend linearly on the period.

Findings

Regret of PUCRL2 scales linearly with the period.

PUCRL2 achieves sub-linear regret with respect to horizon length.

Numerical results confirm the effectiveness of PUCRL2.

Abstract

We study learning in periodic Markov Decision Process(MDP), a special type of non-stationary MDP where both the state transition probabilities and reward functions vary periodically, under the average reward maximization setting. We formulate the problem as a stationary MDP by augmenting the state space with the period index, and propose a periodic upper confidence bound reinforcement learning-2 (PUCRL2) algorithm. We show that the regret of PUCRL2 varies linearly with the period and as sub-linear with the horizon length. Numerical results demonstrate the efficacy of PUCRL2.