Risk-Averse Approximate Dynamic Programming with Quantile-Based Risk Measures

TL;DR

This paper develops data-driven approximate dynamic programming algorithms for risk-averse Markov decision processes using quantile-based risk measures, addressing computational challenges with importance sampling, and demonstrates their effectiveness in energy storage bidding.

Contribution

It introduces novel ADP algorithms for risk-averse MDPs with quantile-based risk measures, incorporating importance sampling to improve efficiency in risk regions.

Findings

Algorithms effectively handle risk-averse decision-making.

Importance sampling improves sampling efficiency in risky regions.

Numerical results show practical applicability in energy storage bidding.

Abstract

In this paper, we consider a finite-horizon Markov decision process (MDP) for which the objective at each stage is to minimize a quantile-based risk measure (QBRM) of the sequence of future costs; we call the overall objective a dynamic quantile-based risk measure (DQBRM). In particular, we consider optimizing dynamic risk measures where the one-step risk measures are QBRMs, a class of risk measures that includes the popular value at risk (VaR) and the conditional value at risk (CVaR). Although there is considerable theoretical development of risk-averse MDPs in the literature, the computational challenges have not been explored as thoroughly. We propose data-driven and simulation-based approximate dynamic programming (ADP) algorithms to solve the risk-averse sequential decision problem. We address the issue of inefficient sampling for risk applications in simulated settings and present a procedure, based on importance sampling, to direct samples toward the "risky region" as the ADP algorithm progresses. Finally, we show numerical results of our algorithms in the context of an application involving risk-averse bidding for energy storage.