Strategy Complexity of Parity Objectives in Countable MDPs

TL;DR

This paper analyzes the complexity of strategies needed for parity objectives in countably infinite MDPs, revealing that simple strategies like 1-bit Markov strategies are often sufficient, unlike in finite MDPs.

Contribution

It provides a complete characterization of the strategy complexity for all subclasses of parity objectives in countably infinite MDPs, including the necessity of various strategy types.

Findings

1-bit Markov strategies suffice for ε-optimal strategies in general parity objectives.

Optimal strategies may require infinite memory and may not always exist.

Strategy complexity depends on the number of colors and branching degree.

Abstract

We study countably infinite MDPs with parity objectives. Unlike in finite MDPs, optimal strategies need not exist, and may require infinite memory if they do. We provide a complete picture of the exact strategy complexity of $\varepsilon$-optimal strategies (and optimal strategies, where they exist) for all subclasses of parity objectives in the Mostowski hierarchy. Either MD-strategies, Markov strategies, or 1-bit Markov strategies are necessary and sufficient, depending on the number of colors, the branching degree of the MDP, and whether one considers $\varepsilon$-optimal or optimal strategies. In particular, 1-bit Markov strategies are necessary and sufficient for $\varepsilon$-optimal (resp. optimal) strategies for general parity objectives.