Continuous-time Risk-sensitive Reinforcement Learning via Quadratic Variation Penalty

TL;DR

This paper introduces a novel risk-sensitive reinforcement learning framework in continuous time that incorporates quadratic variation penalties, enabling more robust decision-making under uncertainty and extending existing algorithms to risk-aware scenarios.

Contribution

It develops a martingale-based characterization of risk-sensitive RL with quadratic variation, adapting existing algorithms for risk sensitivity and proving convergence in financial models.

Findings

The proposed method effectively incorporates risk sensitivity via quadratic variation.

Convergence is proven for the algorithm in Merton's investment problem.

Simulation results show improved finite-sample performance in control tasks.

Abstract

This paper studies continuous-time risk-sensitive reinforcement learning (RL) under the entropy-regularized, exploratory diffusion process formulation with the exponential-form objective. The risk-sensitive objective arises either as the agent's risk attitude or as a distributionally robust approach against the model uncertainty. Owing to the martingale perspective in Jia and Zhou (J Mach Learn Res 24(161): 1--61, 2023) the risk-sensitive RL problem is shown to be equivalent to ensuring the martingale property of a process involving both the value function and the q-function, augmented by an additional penalty term: the quadratic variation of the value process, capturing the variability of the value-to-go along the trajectory. This characterization allows for the straightforward adaptation of existing RL algorithms developed for non-risk-sensitive scenarios to incorporate risk sensitivity by adding the realized variance of the value process. Additionally, I highlight that the conventional policy gradient representation is inadequate for risk-sensitive problems due to the nonlinear nature of quadratic variation; however, q-learning offers a solution and extends to infinite horizon settings. Finally, I prove the convergence of the proposed algorithm for Merton's investment problem and quantify the impact of temperature parameter on the behavior of the learning procedure. I also conduct simulation experiments to demonstrate how risk-sensitive RL improves the finite-sample performance in the linear-quadratic control problem.