Hitting time for Markov decision process

TL;DR

This paper introduces a novel definition of hitting time for Markov decision processes (MDPs) by relating them to Markov processes with matching stationary distributions, addressing limitations of traditional approaches.

Contribution

The authors propose a new method to define MDP hitting time via an associated Markov process whose stationary distribution aligns with the MDP's occupancy measure.

Findings

Established a relationship between MDPs and PageRank.

Constructed a Markov process with matching stationary distribution.

Defined MDP hitting time based on the associated Markov process.

Abstract

We define the hitting time for a Markov decision process (MDP). We do not use the hitting time of the Markov process induced by the MDP because the induced chain may not have a stationary distribution. Even it has a stationary distribution, the stationary distribution may not coincide with the (normalized) occupancy measure of the MDP. We observe a relationship between the MDP and the PageRank. Using this observation, we construct an MP whose stationary distribution coincides with the normalized occupancy measure of the MDP and we define the hitting time of the MDP as the hitting time of the associated MP.