On Bellman's principle with inequality constraints

TL;DR

This paper addresses a paradox in constrained Markov decision processes where Bellman's principle appears violated, and proposes a modified version that accounts for changing constraints at reachable states.

Contribution

It introduces a revised Bellman's principle that accommodates constraint changes in constrained MDPs, resolving previous inconsistencies.

Findings

Modified Bellman's principle preserves optimality with changing constraints.

Demonstrates resolution of Haviv's example of Bellman's principle violation.

Provides a framework for constrained decision processes with dynamic constraints.

Abstract

We consider an example by Haviv (1996) of a constrained Markov decision process that, in some sense, violates Bellman's principle. We resolve this issue by showing how to preserve a form of Bellman's principle that accounts for a change of constraint at states that are reachable from the initial state.