# Forward-PECVaR Algorithm: Exact Evaluation for CVaR SSPs

## TL;DR

This paper introduces Forward-PECVaR, an exact algorithm for evaluating stationary policies in CVaR-SSPs with non-uniform costs, enabling better comparison and analysis of risk-aware policies in stochastic shortest path problems.

## Contribution

The paper proposes Forward-PECVaR, the first exact evaluation algorithm for CVaR-SSPs with non-uniform costs, improving policy assessment and comparison in risk-sensitive decision-making.

## Key findings

- Exact evaluation is crucial for policy comparison.
- Using smaller  and adequate atoms improves approximation quality.
- Empirical results demonstrate the importance of parameter choices.

## Abstract

The Stochastic Shortest Path (SSP) problem models probabilistic sequential-decision problems where an agent must pursue a goal while minimizing a cost function. Because of the probabilistic dynamics, it is desired to have a cost function that considers risk. Conditional Value at Risk (CVaR) is a criterion that allows modeling an arbitrary level of risk by considering the expectation of a fraction $\alpha$ of worse trajectories. Although an optimal policy is non-Markovian, solutions of CVaR-SSP can be found approximately with Value Iteration based algorithms such as CVaR Value Iteration with Linear Interpolation (CVaRVIQ) and CVaR Value Iteration via Quantile Representation (CVaRVILI). These type of solutions depends on the algorithm's parameters such as the number of atoms and $\alpha_0$ (the minimum $\alpha$). To compare the policies returned by these algorithms, we need a way to exactly evaluate stationary policies of CVaR-SSPs. Although there is an algorithm that evaluates these policies, this only works on problems with uniform costs. In this paper, we propose a new algorithm, Forward-PECVaR (ForPECVaR), that evaluates exactly stationary policies of CVaR-SSPs with non-uniform costs. We evaluate empirically CVaR Value Iteration algorithms that found solutions approximately regarding their quality compared with the exact solution, and the influence of the algorithm parameters in the quality and scalability of the solutions. Experiments in two domains show that it is important to use an $\alpha_0$ smaller than the $\alpha$ target and an adequate number of atoms to obtain a good approximation.