# Variance-Based Risk Estimations in Markov Processes via Transformation with State Lumping

## TL;DR

This paper introduces a state-lumping transformation method for variance-based risk estimation in Markov decision processes, improving efficiency and accuracy in risk-sensitive reinforcement learning.

## Contribution

It proposes a novel state lumping technique using isotopic states to reduce the complexity of risk estimation in transformed Markov processes.

## Key findings

- State lumping reduces computational complexity.
- The SAT accurately estimates mean-variance and exponential utility risks.
- Naive reward simplification introduces significant errors.

## Abstract

Variance plays a crucial role in risk-sensitive reinforcement learning, and most risk measures can be analyzed via variance. In this paper, we consider two law-invariant risks as examples: mean-variance risk and exponential utility risk. With the aid of the state-augmentation transformation (SAT), we show that, the two risks can be estimated in Markov decision processes (MDPs) with a stochastic transition-based reward and a randomized policy. To relieve the enlarged state space, a novel definition of isotopic states is proposed for state lumping, considering the special structure of the transformed transition probability. In the numerical experiment, we illustrate state lumping in the SAT, errors from a naive reward simplification, and the validity of the SAT for the two risk estimations.