Finite Optimal Control for Time-Bounded Reachability in CTMDPs and Continuous-Time Markov Games

TL;DR

This paper proves the existence of finite, deterministic, and time-structured optimal control strategies for time-bounded reachability in continuous-time Markov decision processes and games, simplifying their analysis.

Contribution

It establishes the existence of finite, deterministic, and time-partitioned optimal strategies for CTMDPs and extends these results to continuous-time Markov games.

Findings

Optimal strategies are deterministic and timed-positional.

Time can be divided into finite intervals with simple strategies.

Properties extend from CTMDPs to Markov games.

Abstract

We establish the existence of optimal scheduling strategies for time-bounded reachability in continuous-time Markov decision processes, and of co-optimal strategies for continuous-time Markov games. Furthermore, we show that optimal control does not only exist, but has a surprisingly simple structure: The optimal schedulers from our proofs are deterministic and timed-positional, and the bounded time can be divided into a finite number of intervals, in which the optimal strategies are positional. That is, we demonstrate the existence of finite optimal control. Finally, we show that these pleasant properties of Markov decision processes extend to the more general class of continuous-time Markov games, and that both early and late schedulers show this behaviour.