Online Convex Optimization in Adversarial Markov Decision Processes

TL;DR

This paper develops an online learning algorithm for adversarial Markov decision processes with unknown transitions, achieving improved regret bounds by using entropic regularization and extending to convex performance criteria.

Contribution

It introduces a novel online algorithm for adversarial MDPs that handles unknown transitions and convex performance measures, with improved regret bounds.

Findings

Achieves $ ilde{O}(L|X| ext{sqrt}(|A|T))$ regret bound.

Extends adversarial MDP model to convex performance criteria.

Uses entropic regularization for algorithm implementation.

Abstract

We consider online learning in episodic loop-free Markov decision processes (MDPs), where the loss function can change arbitrarily between episodes, and the transition function is not known to the learner. We show $\tilde{O}(L|X|\sqrt{|A|T})$ regret bound, where $T$ is the number of episodes, $X$ is the state space, $A$ is the action space, and $L$ is the length of each episode. Our online algorithm is implemented using entropic regularization methodology, which allows to extend the original adversarial MDP model to handle convex performance criteria (different ways to aggregate the losses of a single episode) , as well as improve previous regret bounds.