Reinforcement Learning with Quasi-Hyperbolic Discounting

TL;DR

This paper introduces the first model-free reinforcement learning algorithm to find Markov Perfect Equilibria under Quasi-Hyperbolic discounting, capturing human-like impatience and improving practical RL applications.

Contribution

It presents a novel model-free algorithm for computing MPE in QH-discounted RL, with theoretical convergence analysis and numerical validation.

Findings

Algorithm converges to MPE if it converges

Numerical validation in inventory system

Advances practical RL with human-like discounting

Abstract

Reinforcement learning has traditionally been studied with exponential discounting or the average reward setup, mainly due to their mathematical tractability. However, such frameworks fall short of accurately capturing human behavior, which has a bias towards immediate gratification. Quasi-Hyperbolic (QH) discounting is a simple alternative for modeling this bias. Unlike in traditional discounting, though, the optimal QH-policy, starting from some time $t_1,$ can be different to the one starting from $t_2.$ Hence, the future self of an agent, if it is naive or impatient, can deviate from the policy that is optimal at the start, leading to sub-optimal overall returns. To prevent this behavior, an alternative is to work with a policy anchored in a Markov Perfect Equilibrium (MPE). In this work, we propose the first model-free algorithm for finding an MPE. Using a two-timescale analysis, we show that, if our algorithm converges, then the limit must be an MPE. We also validate this claim numerically for the standard inventory system with stochastic demands. Our work significantly advances the practical application of reinforcement learning.