Transition-based versus State-based Reward Functions for MDPs with Value-at-Risk

TL;DR

This paper investigates how different reward function representations affect the estimation of Value-at-Risk in Markov decision processes, revealing that simplified reward functions can alter VaR outcomes and proposing methods for accurate estimation.

Contribution

It demonstrates the impact of transition-based versus state-based reward functions on VaR in MDPs and introduces a transformation algorithm for accurate VaR estimation in complex reward settings.

Findings

State-based reward functions can change VaR compared to transition-based ones.

Spectral theory and CLT can estimate VaR in long-horizon MDPs.

A transformation algorithm preserves total reward distribution for VaR estimation.

Abstract

In reinforcement learning, the reward function on current state and action is widely used. When the objective is about the expectation of the (discounted) total reward only, it works perfectly. However, if the objective involves the total reward distribution, the result will be wrong. This paper studies Value-at-Risk (VaR) problems in short- and long-horizon Markov decision processes (MDPs) with two reward functions, which share the same expectations. Firstly we show that with VaR objective, when the real reward function is transition-based (with respect to action and both current and next states), the simplified (state-based, with respect to action and current state only) reward function will change the VaR. Secondly, for long-horizon MDPs, we estimate the VaR function with the aid of spectral theory and the central limit theorem. Thirdly, since the estimation method is for a Markov reward process with the reward function on current state only, we present a transformation algorithm for the Markov reward process with the reward function on current and next states, in order to estimate the VaR function with an intact total reward distribution.