Uniform value in Dynamic Programming

TL;DR

This paper establishes conditions under which a uniform value exists in long-horizon dynamic programming problems, with applications to Markov decision processes and generalizations of prior results.

Contribution

It provides new sufficient conditions for the existence of uniform and limit values in dynamic programming with large horizons, extending previous work.

Findings

Existence of uniform value under precompact state space and uniform continuity

Existence of limit value with non-expansive transition correspondence

Generalizations for Markov decision processes

Abstract

We consider dynamic programming problems with a large time horizon, and give sufficient conditions for the existence of the uniform value. As a consequence, we obtain an existence result when the state space is precompact, payoffs are uniformly continuous and the transition correspondence is non expansive. In the same spirit, we give an existence result for the limit value. We also apply our results to Markov decision processes and obtain a few generalizations of existing results.