Actor-Critic Algorithm for Dynamic Expectile and CVaR

TL;DR

This paper introduces a model-free off-policy actor-critic algorithm for dynamic risk optimization, specifically targeting expectile and CVaR, demonstrating superior risk-averse policy learning in empirical tests.

Contribution

It develops a novel surrogate policy gradient method without transition perturbation and new value learning techniques for dynamic risk measures.

Findings

The algorithm effectively learns risk-averse policies.

It outperforms existing methods in risk-averse domains.

Empirical results validate the approach's effectiveness.

Abstract

Optimizing dynamic risk with stochastic policies is challenging in both policy updates and value learning. The former typically requires transition perturbation, while the latter may rely on model-based approaches. To address these challenges, we propose a surrogate policy gradient without transition perturbation under softmax policy parameterization. We further develop model-free value learning methods for dynamic expectile and conditional value-at-risk by leveraging elicitability. Finally, inspired by Expected SARSA and Expected Policy Gradient, a model-free off-policy actor-critic algorithm is constructed. Empirical results in domains with verifiable risk-averse behavior show that our algorithm can learn risk-averse policy and consistently outperforms other existing methods.