Research on Optimal Control Problem Based on Reinforcement Learning under Knightian Uncertainty

TL;DR

This paper develops a framework for optimal control in reinforcement learning under Knightian uncertainty, deriving the HJB equation, analyzing Gaussian policies in LQ cases, and validating results with simulations.

Contribution

It introduces a novel approach to stochastic control under Knightian uncertainty, including deriving the HJB equation and analyzing Gaussian optimal policies in LQ scenarios.

Findings

Optimal randomized feedback control follows a Gaussian distribution.

Knightian uncertainty influences the variance of the optimal policy.

Theoretical results are validated through numerical simulations.

Abstract

Considering that the decision-making environment faced by reinforcement learning (RL) agents is full of Knightian uncertainty, this paper describes the exploratory state dynamics equation in Knightian uncertainty to study the entropy-regularized relaxed stochastic control problem in a Knightian uncertainty environment. By employing stochastic analysis theory and the dynamic programming principle under nonlinear expectation, we derive the Hamilton-Jacobi-Bellman (HJB) equation and solve for the optimal policy that achieves a trade-off between exploration and exploitation. Subsequently, for the linear-quadratic (LQ) case, we examine the agent's optimal randomized feedback control under both state-dependent and state-independent reward scenarios, proving that the optimal randomized feedback control follows a Gaussian distribution in the LQ framework. Furthermore, we investigate how the degree of Knightian uncertainty affects the variance of the optimal feedback policy. Additionally, we establish the solvability equivalence between non-exploratory and exploratory LQ problems under Knightian uncertainty and analyze the associated exploration cost. Finally, we provide an LQ example and validate the theoretical findings through numerical simulations.