Relaxed Equilibria for Time-Inconsistent Markov Decision Processes

TL;DR

This paper establishes the existence of relaxed equilibrium policies in infinite-horizon, time-inconsistent MDPs with general discount functions, using entropy regularization and weak convergence, without requiring convexity assumptions.

Contribution

It introduces a novel existence proof for relaxed equilibria in time-inconsistent MDPs via entropy regularization, accommodating randomized policies without convexity constraints.

Findings

Existence of relaxed equilibrium in entropy-regularized MDPs.

Approximation of original MDPs by entropy-regularized versions as regularization diminishes.

Randomized policies enable equilibrium existence without convexity assumptions.

Abstract

This paper considers an infinite-horizon Markov decision process (MDP) that allows for general non-exponential discount functions, in both discrete and continuous time. Due to the inherent time inconsistency, we look for a randomized equilibrium policy (i.e., relaxed equilibrium) in an intra-personal game between an agent's current and future selves. When we modify the MDP by entropy regularization, a relaxed equilibrium is shown to exist by a nontrivial entropy estimate. As the degree of regularization diminishes, the entropy-regularized MDPs approximate the original MDP, which gives the general existence of a relaxed equilibrium in the limit by weak convergence arguments. As opposed to prior studies that consider only deterministic policies, our existence of an equilibrium does not require any convexity (or concavity) of the controlled transition probabilities and reward function. Interestingly, this benefit of considering randomized policies is unique to the time-inconsistent case.