Policy Iteration Achieves Regularized Equilibrium under Time Inconsistency

TL;DR

This paper introduces a policy iteration algorithm for entropy-regularized, time-inconsistent stochastic control problems, proving its exponential convergence to an equilibrium policy via a novel PDE approach.

Contribution

It develops a new policy iteration method for time-inconsistent control problems and proves its exponential convergence to equilibrium, addressing the existence and uniqueness of solutions.

Findings

Proves exponential convergence of the policy iteration algorithm.

Establishes the existence and uniqueness of classical solutions to the EEHJB equation.

Provides a constructive proof of equilibrium policy existence in complex stochastic control.

Abstract

For a general entropy-regularized time-inconsistent stochastic control problem, we propose a policy iteration algorithm (PIA) and establish its convergence to an equilibrium policy with an exponential convergence rate. The design of the PIA is based on a coupled system of non-local partial differential equations, called the exploratory equilibrium Hamilton--Jacobi--Bellman (EEHJB) equation. As opposed to the standard time-consistent case, policy improvement fails in general and the target value function (now an equilibrium value function) is not even known to exist a priori. To overcome these, we prove that the value functions generated by the PIA form a Cauchy sequence in a specialized Banach space, hence admit a limit, and the rate of convergence is exponential, on the strength of the Bismut--Elworthy--Li formula of stochastic representation. The limiting value function is shown to fulfill the EEHJB equation, which induces an equilibrium policy in a Gibbs form. Such convergence in value additionally implies uniform convergence of the generated policies to the equilibrium policy, again with an exponential rate. As a byproduct, the PIA gives a constructive proof of the global existence and uniqueness of a classical solution to our general EEHJB equation, whose well-posedness has not been explored in the literature.