General limit value in Dynamic Programming

TL;DR

This paper establishes conditions under which a unique limit value exists in dynamic programming problems as decision-makers become infinitely patient, unifying various payoff models and providing a comprehensive theoretical framework.

Contribution

It introduces a general condition for the uniform convergence of value functions in dynamic programming as patience tends to infinity, identifying a unique limit value independent of evaluation sequences.

Findings

Uniform convergence occurs iff the sequence of value functions is totally bounded.

A unique limit value function $v^*$ exists, independent of the evaluation sequence.

The results apply to discounted, average, and stochastic transition models.

Abstract

We consider a dynamic programming problem with arbitrary state space and bounded rewards. Is it possible to define in an unique way a limit value for the problem, where the "patience" of the decision-maker tends to infinity ? We consider, for each evaluation $\theta$ (a probability distribution over positive integers) the value function $v_{\theta}$ of the problem where the weight of any stage $t$ is given by $\theta_t$, and we investigate the uniform convergence of a sequence $(v_{\theta^k})_k$ when the "impatience" of the evaluations vanishes, in the sense that $\sum_{t} |\theta^k_{t}-\theta^k_{t+1}| \rightarrow_{k \to \infty} 0$. We prove that this uniform convergence happens if and only if the metric space ${v_{\theta^k}, k\geq 1}$ is totally bounded. Moreover there exists a particular function $v^*$, independent of the particular chosen sequence $({\theta^k})_k$, such that any limit point of such sequence of value functions is precisely $v^*$. Consequently, while speaking of uniform convergence of the value functions, $v^*$ may be considered as the unique possible limit when the patience of the decision-maker tends to infinity. The result applies in particular to discounted payoffs when the discount factor vanishes, as well as to average payoffs where the number of stages goes to infinity, and also to models with stochastic transitions. We present tractable corollaries, and we discuss counterexamples and a conjecture.