On reachability of Markov chains: A long-run average approach

TL;DR

This paper studies reachability and controllability in discrete-time Markov control models with countable state and action spaces, using long-run average reward functions to characterize domains and optimize reachability probabilities.

Contribution

It introduces a value function-based approach to characterize attraction domains and solve reachability problems, including constrained variants, with linear programming in finite cases.

Findings

Characterized domain of attraction and escape set using value functions

Solved reachability and avoidance probability maximization problems

Applied results to a stochastic navigation example

Abstract

We consider a Markov control model in discrete time with countable both state space and action space. Using the value function of a suitable long-run average reward problem, we study various reachability/controllability problems. First, we characterize the domain of attraction and escape set of the system, and a generalization called $p$-domain of attraction, using the aforementioned value function. Next, we solve the problem of maximizing the probability of reaching a set $A$ while avoiding a set $B$. Finally, we consider a constrained version of the previous problem where we ask for the probability of reaching the set $B$ to be bounded. In the finite case, we use linear programming formulations to solve these problems. Finally, we apply our results to a example of an object that navigates under stochastic influence.