Strategy Complexity of Point Payoff, Mean Payoff and Total Payoff Objectives in Countable MDPs
Richard Mayr, Eric Munday

TL;DR
This paper analyzes the strategy complexity in countably infinite MDPs for different payoff objectives, identifying the memory requirements for optimal and near-optimal strategies across point, mean, and total payoffs.
Contribution
It provides a complete characterization of the memory needed for strategies to optimize various payoff objectives in countable MDPs.
Findings
Memoryless strategies suffice in some cases.
Step counters or reward counters are needed in others.
The results clarify the strategy complexity landscape for these objectives.
Abstract
We study countably infinite Markov decision processes (MDPs) with real-valued transition rewards. Every infinite run induces the following sequences of payoffs: 1. Point payoff (the sequence of directly seen transition rewards), 2. Mean payoff (the sequence of the sums of all rewards so far, divided by the number of steps), and 3. Total payoff (the sequence of the sums of all rewards so far). For each payoff type, the objective is to maximize the probability that the is non-negative. We establish the complete picture of the strategy complexity of these objectives, i.e., how much memory is necessary and sufficient for -optimal (resp. optimal) strategies. Some cases can be won with memoryless deterministic strategies, while others require a step counter, a reward counter, or both.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
