Exploratory Randomization for Discrete-Time Risk-Sensitive Benchmarked Investment Management with Reinforcement Learning

TL;DR

This paper introduces a novel exploration mechanism for risk-sensitive portfolio management using reinforcement learning, combining stochastic control, duality, and fractional Kelly strategies to improve investment policies.

Contribution

It develops a tractable, risk-sensitive portfolio model with endogenous regularization, connecting RL and control theory to guide policy-gradient methods.

Findings

Analytical solution for risk-sensitive control problem

Bounds on exploration based on risk and asset covariance

Interpretation of strategies via fractional Kelly approaches

Abstract

This paper bridges reinforcement learning (RL) and risk-sensitive stochastic control by introducing a tractable exploration mechanism for policy search in risk-sensitive portfolio management, with known and unknown model parameters, that yields an endogenous relative-entropy regularization. We construct a discrete-time risk-sensitive benchmarked investment model. This model combines a factor-based asset universe with periodic portfolio rebalancing. Exploration is incorporated through user-specified Gaussian perturbations to baseline (exploitative) controls. The risk-sensitive stochastic control problem is solved analytically using the Free Energy-Entropy Duality. The Duality recasts the control problem as a linear-quadratic-Gaussian game and introduces a natural penalty for exploration. This approach yields simple sufficiency conditions for optimality. It also induces intuitive bounds on exploration based on risk sensitivity, asset covariance, and rebalancing frequency. Additionally, the optimal investment strategy can be interpreted through the lens of fractional Kelly strategies. By connecting risk-sensitive control theory and RL, this work provides a principled parametric family for policy-gradient implementations, guiding the design of RL methods.