Reinforcement Learning with Dynamic Convex Risk Measures

TL;DR

This paper introduces a model-free reinforcement learning framework that incorporates dynamic convex risk measures for time-consistent risk-sensitive decision making, demonstrated across finance and robotics applications.

Contribution

It develops a novel RL approach using dynamic programming and policy gradients to handle risk measures, with an actor-critic neural network implementation for practical optimization.

Findings

Effective in financial trading and hedging tasks

Robust obstacle avoidance in robotics

Flexible across multiple risk-sensitive applications

Abstract

We develop an approach for solving time-consistent risk-sensitive stochastic optimization problems using model-free reinforcement learning (RL). Specifically, we assume agents assess the risk of a sequence of random variables using dynamic convex risk measures. We employ a time-consistent dynamic programming principle to determine the value of a particular policy, and develop policy gradient update rules that aid in obtaining optimal policies. We further develop an actor-critic style algorithm using neural networks to optimize over policies. Finally, we demonstrate the performance and flexibility of our approach by applying it to three optimization problems: statistical arbitrage trading strategies, financial hedging, and obstacle avoidance robot control.