Variational Regret Bounds for Reinforcement Learning

TL;DR

This paper introduces a novel algorithm for non-stationary reinforcement learning in MDPs, providing the first variational regret bounds that adapt to changing reward functions and transition probabilities over time.

Contribution

It presents the first variational regret bounds for reinforcement learning in non-stationary MDPs, linking regret to the total variation in the environment.

Findings

Regret bounds scale with total variation in MDPs.

Algorithm guarantees performance in changing environments.

First variational regret bounds for general RL setting.

Abstract

We consider undiscounted reinforcement learning in Markov decision processes (MDPs) where both the reward functions and the state-transition probabilities may vary (gradually or abruptly) over time. For this problem setting, we propose an algorithm and provide performance guarantees for the regret evaluated against the optimal non-stationary policy. The upper bound on the regret is given in terms of the total variation in the MDP. This is the first variational regret bound for the general reinforcement learning setting.