A Subgame Perfect Equilibrium Reinforcement Learning Approach to Time-inconsistent Problems

TL;DR

This paper introduces a novel reinforcement learning framework called SPERL for solving time-inconsistent problems by establishing subgame perfect equilibrium, addressing key challenges with a new backward policy iteration algorithm and demonstrating its effectiveness in portfolio optimization.

Contribution

The paper develops the SPERL framework and the BPI algorithm, extending dynamic programming to handle time-inconsistency in RL, which was previously unaddressed.

Findings

BPI effectively solves TIC problems in RL.

The framework demonstrates convergence in portfolio selection.

Model identifiability is improved with the new approach.

Abstract

In this paper, we establish a subgame perfect equilibrium reinforcement learning (SPERL) framework for time-inconsistent (TIC) problems. In the context of RL, TIC problems are known to face two main challenges: the non-existence of natural recursive relationships between value functions at different time points and the violation of Bellman's principle of optimality that raises questions on the applicability of standard policy iteration algorithms for unprovable policy improvement theorems. We adapt an extended dynamic programming theory and propose a new class of algorithms, called backward policy iteration (BPI), that solves SPERL and addresses both challenges. To demonstrate the practical usage of BPI as a training framework, we adapt standard RL simulation methods and derive two BPI-based training algorithms. We examine our derived training frameworks on a mean-variance portfolio selection problem and evaluate some performance metrics including convergence and model identifiability.