Strategy complexity of finite-horizon Markov decision processes and simple stochastic games

TL;DR

This paper analyzes the strategy complexity of finite-horizon MDPs and SSGs, providing asymptotically optimal bounds on memory requirements and revealing sub-exponential lower bounds on the period of optimal strategies.

Contribution

It establishes tight bounds on the memory size needed for strategies in finite-horizon MDPs and SSGs and investigates the periodic properties of optimal strategies.

Findings

Counter-based strategies require at most log log (1/ε) + n+1 memory states.

Memory of size Ω(log log (1/ε) + n) is necessary.

Optimal strategies have a sub-exponential lower bound on their period.

Abstract

Markov decision processes (MDPs) and simple stochastic games (SSGs) provide a rich mathematical framework to study many important problems related to probabilistic systems. MDPs and SSGs with finite-horizon objectives, where the goal is to maximize the probability to reach a target state in a given finite time, is a classical and well-studied problem. In this work we consider the strategy complexity of finite-horizon MDPs and SSGs. We show that for all $\epsilon>0$, the natural class of counter-based strategies require at most $\log \log (\frac{1}{\epsilon}) + n+1$ memory states, and memory of size $\Omega(\log \log (\frac{1}{\epsilon}) + n)$ is required. Thus our bounds are asymptotically optimal. We then study the periodic property of optimal strategies, and show a sub-exponential lower bound on the period for optimal strategies.