Online Markov decision processes with policy iteration

TL;DR

This paper introduces practical online MDP algorithms with policy iteration that adapt to changing rewards, offering theoretical guarantees and compatibility with function approximation for large state spaces.

Contribution

The paper presents a novel online MDP algorithm with policy iteration that achieves sublinear regret and can be combined with function approximation for large-scale problems.

Findings

The proposed algorithm achieves sublinear regret bounds.

It can be integrated with function approximation for large state spaces.

Experimental results demonstrate its practical usefulness.

Abstract

The online Markov decision process (MDP) is a generalization of the classical Markov decision process that incorporates changing reward functions. In this paper, we propose practical online MDP algorithms with policy iteration and theoretically establish a sublinear regret bound. A notable advantage of the proposed algorithm is that it can be easily combined with function approximation, and thus large and possibly continuous state spaces can be efficiently handled. Through experiments, we demonstrate the usefulness of the proposed algorithm.