Continuous-time q-Learning for Jump-Diffusion Models under Tsallis Entropy

TL;DR

This paper develops continuous-time q-learning algorithms for jump-diffusion models using Tsallis entropy, providing explicit policy characterizations and demonstrating their effectiveness in financial and control problems.

Contribution

It introduces novel q-learning algorithms under Tsallis entropy in continuous time, including explicit policy characterization and actor-critic methods for jump-diffusion models.

Findings

Optimal policies are explicitly characterized as distributions with compact support.

The proposed algorithms perform well in dark pool liquidation and control problems.

Tsallis entropy regularization leads to non-Gibbs optimal policies.

Abstract

This paper studies the continuous-time reinforcement learning in jump-diffusion models by featuring the q-learning (the continuous-time counterpart of Q-learning) under Tsallis entropy regularization. Contrary to the Shannon entropy, the general form of Tsallis entropy renders the optimal policy not necessarily a Gibbs measure. Herein, the Lagrange multiplier and KKT condition are needed to ensure that the learned policy is a probability density function. As a consequence, the characterization of the optimal policy using the q-function also involves a Lagrange multiplier. In response, we establish the martingale characterization of the q-function and devise two q-learning algorithms depending on whether the Lagrange multiplier can be derived explicitly or not. In the latter case, we consider different parameterizations of the optimal q-function and the optimal policy, and update them alternatively in an Actor-Critic manner. We also study two numerical examples, namely, an optimal liquidation problem in dark pools and a non-LQ control problem. It is interesting to see therein that the optimal policies under the Tsallis entropy regularization can be characterized explicitly, which are distributions concentrated on some compact support. The satisfactory performance of our q-learning algorithms is illustrated in each example.