Risk-Sensitive Q-Learning in Continuous Time with Application to Dynamic Portfolio Selection

TL;DR

This paper introduces a risk-sensitive reinforcement learning framework in continuous time, proving optimal policies for certain functionals and proposing a novel q-learning algorithm, demonstrated through a portfolio management simulation.

Contribution

It develops a new continuous-time risk-sensitive q-learning algorithm and characterizes optimal policies for risk-sensitive objectives involving nonlinear functionals.

Findings

The proposed CT-RS-q algorithm effectively learns risk-sensitive policies.

Optimal policies are Markovian with respect to an augmented environment.

Simulation results show improved portfolio management performance.

Abstract

This paper studies the problem of risk-sensitive reinforcement learning (RSRL) in continuous time, where the environment is characterized by a controllable stochastic differential equation (SDE) and the objective is a potentially nonlinear functional of cumulative rewards. We prove that when the functional is an optimized certainty equivalent (OCE), the optimal policy is Markovian with respect to an augmented environment. We also propose \textit{CT-RS-q}, a risk-sensitive q-learning algorithm based on a novel martingale characterization approach. Finally, we run a simulation study on a dynamic portfolio selection problem and illustrate the effectiveness of our algorithm.