Strategy Complexity of Limsup and Liminf Threshold Objectives in Countable MDPs, with Applications to Optimal Expected Payoffs

TL;DR

This paper analyzes the strategy complexity in countable Markov decision processes for $ ext{limsup}$ and $ ext{liminf}$ threshold objectives, providing bounds on memory requirements and solving open problems related to optimal expected payoffs.

Contribution

It establishes the complete strategy complexity bounds for $ ext{limsup}$ and $ ext{liminf}$ objectives in countable MDPs and applies these results to open problems on optimal strategies for expected payoffs.

Findings

Bounds on memory requirements for $ ext{limsup}$ and $ ext{liminf}$ objectives.

Complete characterization of strategy complexity in countable MDPs.

Resolution of open problems on optimal strategies for expected $ ext{limsup}$ and $ ext{liminf}$ payoffs.

Abstract

We study Markov decision processes (MDPs) with a countably infinite number of states. The $\limsup$ (resp. $\liminf$) threshold objective is to maximize the probability that the $\limsup$ (resp. $\liminf$) of the infinite sequence of directly seen rewards is non-negative. We establish the complete picture of the strategy complexity of these objectives, i.e., the upper and lower bounds on the memory required by $\varepsilon$-optimal (resp. optimal) strategies. We then apply these results to solve two open problems from (Sudderth, Decisions in Economics and Finance, 2020) about the strategy complexity of optimal strategies for the expected $\limsup$ (resp. $\liminf$) payoff.