MDPs with Setwise Continuous Transition Probabilities

TL;DR

This paper investigates the structure of optimal policies in infinite-state Markov Decision Processes with setwise continuous transition probabilities, accommodating noncompact action sets and various cost criteria, using a new optimal selection theorem.

Contribution

Introduces a novel optimal selection theorem for inf-compact functions and applies it to analyze optimal policies in complex MDPs with setwise continuous transitions.

Findings

Characterizes optimal policies under setwise continuity.

Handles noncompact action sets in infinite-state MDPs.

Provides a unified approach for discounted, undiscounted, and average costs.

Abstract

This paper describes the structure of optimal policies for infinite-state Markov Decision Processes with setwise continuous transition probabilities. The action sets may be noncompact. The objective criteria are either the expected total discounted and undiscounted costs or average costs per unit time. The analysis of optimality equations and inequalities is based on the optimal selection theorem for inf-compact functions introduced in this paper.