On Convergence of Value Iteration for a Class of Total Cost Markov Decision Processes

TL;DR

This paper investigates the convergence properties of value iteration in a broad class of total cost Markov decision processes with arbitrary costs, extending known results and introducing new convergence conditions.

Contribution

It extends convergence results of value iteration to the General Convergence (GC) total cost model, including cases with arbitrary costs and partial convergence scenarios.

Findings

Value iteration converges from above for certain functions.

Transfinite value iteration converges when the optimal cost is nonnegative.

Partial convergence of value iteration occurs for some initial states.

Abstract

We consider a general class of total cost Markov decision processes (MDP) in which the one-stage costs can have arbitrary signs, but the sum of the negative parts of the one-stage costs is finite for all policies and all initial states. We refer to this class as the General Convergence (GC for short) total cost model, and we study the convergence of value iteration for the GC model, in the Borel MDP framework with universally measurable policies. Our main results include: (i) convergence of value iteration when starting from certain functions above the optimal cost function; (ii) convergence of transfinite value iteration starting from zero, in the special case where the optimal cost function is nonnegative; and (iii) partial convergence of value iteration starting from zero, for a subset of initial states. These results extend several previously known results about the convergence of value iteration for either positive costs problems or GC total cost problems. In particular, the first result on convergence of value iteration from above extends a theorem of van der Wal for the GC model. The second result relates to Maitra and Sudderth's analysis of transfinite value iteration for the positive costs model. It suggests connections between the two total cost models when the optimal cost function is nonnegative, and it leads to additional results on the convergence of ordinary non-transfinite value iteration, with a suitably defined dynamic programming operator, for finite state or finite control GC problems. The third result on partial convergence of value iteration is motivated by Whittle's bridging condition for the positive costs model, and provides a novel extension of the bridging condition to the GC model, where there are no sign constraints on the costs.