Strategy Complexity of Mean Payoff, Total Payoff and Point Payoff Objectives in Countable MDPs

TL;DR

This paper analyzes the complexity of strategies needed to optimize different payoff objectives in countably infinite MDPs, revealing varying memory requirements for near-optimal strategies.

Contribution

It provides a complete characterization of the memory complexity for maximizing point, total, and mean payoffs in countable MDPs, including cases requiring complex strategies.

Findings

Memoryless strategies suffice in some cases.

Step counters or reward counters are needed in others.

The paper maps out the exact strategy complexity landscape.

Abstract

We study countably infinite Markov decision processes (MDPs) with real-valued transition rewards. Every infinite run induces the following sequences of payoffs: 1. Point payoff (the sequence of directly seen transition rewards), 2. Total payoff (the sequence of the sums of all rewards so far), and 3. Mean payoff. For each payoff type, the objective is to maximize the probability that the $\liminf$ is non-negative. We establish the complete picture of the strategy complexity of these objectives, i.e., how much memory is necessary and sufficient for $\varepsilon$-optimal (resp. optimal) strategies. Some cases can be won with memoryless deterministic strategies, while others require a step counter, a reward counter, or both.