Regular Policies in Abstract Dynamic Programming

TL;DR

This paper introduces the concept of regular policies in complex dynamic programming models, showing they have better-behaved properties and can improve analysis and algorithms for various challenging problems.

Contribution

It proposes the notion of regular policies, providing a unifying framework to analyze and solve complex undiscounted dynamic programming models.

Findings

Optimal costs over regular policies can have better properties.

Regular policies facilitate analysis of complex models.

Methodology applies to stochastic, minimax, and risk-sensitive problems.

Abstract

We consider challenging dynamic programming models where the associated Bellman equation, and the value and policy iteration algorithms commonly exhibit complex and even pathological behavior. Our analysis is based on the new notion of regular policies. These are policies that are well-behaved with respect to value and policy iteration, and are patterned after proper policies, which are central in the theory of stochastic shortest path problems. We show that the optimal cost function over regular policies may have favorable value and policy iteration properties, which the optimal cost function over all policies need not have. We accordingly develop a unifying methodology to address long standing analytical and algorithmic issues in broad classes of undiscounted models, including stochastic and minimax shortest path problems, as well as positive cost, negative cost, risk-sensitive, and multiplicative cost problems.