Reinforcement Learning with Markov Risk Measures and Multipattern Risk Approximation

TL;DR

This paper introduces mini-batch Markov coherent risk measures and multipattern risk-averse problems, developing a feature-based Q-learning method with regret bounds and practical variants, demonstrated on stochastic and bandit problems.

Contribution

It proposes a novel class of risk measures and a Q-learning algorithm with theoretical guarantees for risk-averse Markov decision processes.

Findings

High-probability regret bound of O(H^2 N^H √K) for the proposed method.

Economical Q-learning variant streamlining policy evaluation.

Empirical validation on stochastic assignment and multi-armed bandit problems.

Abstract

For a risk-averse finite-horizon Markov Decision Problem, we introduce a special class of Markov coherent risk measures, called mini-batch measures. We also define the class of multipattern risk-averse problems that generalizes the class of linear systems. We use both concepts in a feature-based $Q$-learning method with multipattern $Q$-factor approximation and we prove a high-probability regret bound of $\mathcal{O}\big(H^2 N^H \sqrt{ K}\big)$, where $H$ is the horizon, $N$ is the mini-batch size, and $K$ is the number of episodes. We also propose an economical version of the $Q$-learning method that streamlines the policy evaluation (backward) step. The theoretical results are illustrated on a stochastic assignment problem and a short-horizon multi-armed bandit problem.