Optimal Investment and Entropy-Regularized Learning Under Stochastic Volatility Models with Portfolio Constraints

TL;DR

This paper develops a continuous-time reinforcement learning framework for optimal portfolio selection under stochastic volatility, employing entropy regularization and PDE analysis to derive implementable policies.

Contribution

It introduces a novel entropy-regularized control approach with PDE-based analysis for portfolio optimization under stochastic volatility with constraints.

Findings

Derived the entropy-regularized HJB equation for portfolio control.

Proved existence of classical solutions under certain conditions.

Provided a PDE-based interpretation of actor-critic learning dynamics.

Abstract

We study the problem of optimal portfolio selection under stochastic volatility within a continuous time reinforcement learning framework with portfolio constraints. Exploration is modeled through entropy-regularized relaxed controls, where the investor selects probability distributions over admissible portfolio allocations rather than deterministic strategies. Using dynamic programming arguments, we derive the associated entropy-regularized Hamilton-Jacobi-Bellman equation, whose Hamiltonian involves optimization over probability measures supported on a compact control set. We show that the optimal exploratory policy takes the form of a truncated Gaussian distribution characterized by spatial derivatives of the solution of the resulting nonlinear quasilinear parabolic partial differential equation. Under suitable structural conditions on the model coefficients, we prove the existence of classical solutions to this nonlinear HJB equation for the value function. We then establish a verification theorem and analyze the policy-improvement structure induced by the entropy-regularized Hamiltonian, showing how the resulting sequence of PDEs provides a continuous-time interpretation of actor-critic learning dynamics. Finally, our PDE analysis with a semi-closed form of optimal value and optimal policy enables the design of an implementable reinforcement learning algorithm by recasting the optimal problem in a martingale framework.