Exploratory Utility Maximization Problem with Tsallis Entropy

TL;DR

This paper explores utility maximization with Tsallis entropy regularization in a complete market, revealing conditions for well-posedness, characterizing optimal strategies, and demonstrating reinforcement learning advantages through numerical experiments.

Contribution

It introduces Tsallis entropy into utility maximization, analyzes well-posedness issues, and provides semi-closed-form solutions for specific cases, extending classical models.

Findings

Optimal strategies include Gaussian and Wigner semicircle distributions.

Well-posedness depends on the primary temperature function.

Reinforcement learning effectively approximates optimal policies.

Abstract

We study expected utility maximization problem with constant relative risk aversion utility function in a complete market under the reinforcement learning framework. To induce exploration, we introduce the Tsallis entropy regularizer, which generalizes the commonly used Shannon entropy. Unlike the classical Merton's problem, which is always well-posed and admits closed-form solutions, we find that the utility maximization exploratory problem is ill-posed in certain cases, due to over-exploration. With a carefully selected primary temperature function, we investigate two specific examples, for which we fully characterize their well-posedness and provide semi-closed-form solutions. It is interesting to find that one example has the well-known Gaussian distribution as the optimal strategy, while the other features the rare Wigner semicircle distribution, which is equivalent to a scaled Beta distribution. The means of the two optimal exploratory policies coincide with that of the classical counterpart. In addition, we examine the convergence of the value function and optimal exploratory strategy as the exploration vanishes. Finally, we design a reinforcement learning algorithm and conduct numerical experiments to demonstrate the advantages of reinforcement learning.