First Passage Value

TL;DR

This paper extends the concept of Mean First Passage Time to a more general framework called Mean First Passage Value, providing bounds and demonstrating its effectiveness in analyzing complex, metastable stochastic systems, especially Markov Decision Processes.

Contribution

It introduces the Mean First Passage Value (MFPV), generalizes bounds for First Passage Value, and shows that system-wide MFPT can be captured using the second largest eigenvalue in complex systems.

Findings

System-wide MFPT can be approximated using the second largest eigenvalue.

MFPV can represent various quantities like energy, distance, or time.

Bounds on FPV are provided for specified confidence levels.

Abstract

For many stochastic dynamic systems, the Mean First Passage Time (MFPT) is a useful concept, which gives expected time before a state of interest. This work is an extension of MFPT in several ways. (1) We show that for some systems the system-wide MFPT, calculated using the second largest eigenvalue only, captures most of the fundamental dynamics, even for quite complex, high-dimensional systems. (2) We generalize MFPT to Mean First Passage Value (MFPV), which gives a more general value of interest, e.g., energy expenditure, distance, or time. (3) We provide bounds on First Passage Value (FPV) for a given confidence level. At the heart of this work, we emphasize that for our goals, many hybrid systems can be approximated as Markov Decision Processes. So, many systems can be controlled effectively using this framework. However, our framework is particularly useful for metastable systems. Such systems exhibit interesting long-living behaviors from which they are guaranteed to inevitably escape (e.g., eventually arriving at a distinct failure or success state). Our goal is then either minimizing or maximizing the value until escape, depending on the application.