Exploratory Control with Tsallis Entropy for Latent Factor Models

TL;DR

This paper introduces a novel optimal control framework using Tsallis Entropy for latent factor models, deriving $q$-Gaussian optimal distributions and developing a model-agnostic policy applicable to areas like trading strategies.

Contribution

It presents a new approach to optimal control with exploration driven by Tsallis Entropy, deriving explicit solutions in both discrete and continuous time settings.

Findings

Optimal distributions are $q$-Gaussian, characterized by solutions to FBS$ riangle$E and FBSDE.

The method links exploration with standard control solutions, enhancing robustness.

Develops a soft $Q$-learning inspired policy for model-agnostic applications.

Abstract

We study optimal control in models with latent factors where the agent controls the distribution over actions, rather than actions themselves, in both discrete and continuous time. To encourage exploration of the state space, we reward exploration with Tsallis Entropy and derive the optimal distribution over states - which we prove is $q$-Gaussian distributed with location characterized through the solution of an FBS$\Delta$E and FBSDE in discrete and continuous time, respectively. We discuss the relation between the solutions of the optimal exploration problems and the standard dynamic optimal control solution. Finally, we develop the optimal policy in a model-agnostic setting along the lines of soft $Q$-learning. The approach may be applied in, e.g., developing more robust statistical arbitrage trading strategies.