Statistical analysis of the first passage path ensemble of jump processes

TL;DR

This paper develops a statistical framework for analyzing the first passage paths of jump processes, including non-ergodic and non-Markovian cases, with algorithms for practical computation and applications across various fields.

Contribution

It introduces a novel approach to decompose first passage paths into segments and provides algorithms for their statistical analysis, applicable to a broad class of jump processes.

Findings

Applicable to non-ergodic jump processes

Provides efficient algorithms for path statistics

Links first passage analysis with transition path theory

Abstract

The transition mechanism of jump processes between two different subsets in state space reveals important dynamical information of the processes and therefore has attracted considerable attention in the past years. In this paper, we study the first passage path ensemble of both discrete-time and continuous-time jump processes on a finite state space. The main approach is to divide each first passage path into nonreactive and reactive segments and to study them separately. The analysis can be applied to jump processes which are non-ergodic, as well as continuous-time jump processes where the waiting time distributions are non-exponential. In the particular case that the jump processes are both Markovian and ergodic, our analysis elucidates the relations between the study of the first passage paths and the study of the transition paths in transition path theory. We provide algorithms to numerically compute statistics of the first passage path ensemble. The computational complexity of these algorithms scales with the complexity of solving a linear system, for which efficient methods are available. Several examples demonstrate the wide applicability of the derived results across research areas.

Figures

Tables

Table 1. Table 1: Notations used throughout the paper

${𝒩}_x$	set of neighbor nodes	$𝒯$	set of sink nodes
$V^{-}$	subset of node set $V$	$V^{+}$	subset of node set $V$
$q$	committor function	$q^{-}$	backward committor function
$m$	invariant measure of discrete process	$Z$	normalization constant
${\Xi }_{non}$	nonreactive trajectory ensemble	${\Xi }_r$	reactive trajectory ensemble
${\sigma }_x$	last hitting time of set $A$	$\mu$	initial distribution on set $A$
${\mu }_r$	initial distribution of reactive trajectories	$p$	transition probability of discrete process
$\psi$	probability density of the waiting time along an edge	$\overline{p}$	transition probability of the nonreactive trajectories
$\theta$	average number of times that the first passage paths visit a node	$\tilde{p}$	transition probability of the reactive trajectories
${\overline{\theta }}^{\prime }$	average number of times that nonreactive trajectories (except the last node) visit a node	$J$	average number of times that the first passage paths visit an edge
$\overline{\theta }$	average number of times that nonreactive trajectories visit a node	$\overline{J}$	average number of times that nonreactive trajectories visit an edge
$\tilde{\theta }$	average number of times that reactive trajectories visit a node	$\tilde{J}$	average number of times that the reactive trajectories visit an edge
$a$	probability that system stay at a node longer than certain amount of time	$T$	average total time of the first passage paths
$b$	probability density that the system jumps along an edge at a certain time	$\overline{T}$	average total time of the nonreactive trajectories
$\kappa$	average amount of time that the system stays at a node	$\tilde{T}$	average total time of the reactive trajectories

Table 2. Table 2: Example 1 1 1 . Ensemble averages of the continuous-time jump processes and the associated discrete-time Markov jump processes are displayed for each node, when the jump times obey either exponential distribution, power law distribution, or Weibull distribution. κ 𝜅 \kappa is defined in ( 44 ). θ ¯ ′ superscript ¯ 𝜃 ′ \bar{\theta}^{\prime} , T ¯ ¯ 𝑇 \bar{T} are related to the nonreactive trajectories and are defined in ( 18 ), ( 57 ). θ ~ ~ 𝜃 \widetilde{\theta} , T ~ ~ 𝑇 \widetilde{T} are related to the reactive trajectories and are defined in ( 27 ), ( 60 ). θ 𝜃 \theta , T 𝑇 T are related to the whole mean first passage paths and are defined in ( 35 ), ( 63 ). For each ensemble average, the column with label “Total” shows the sum of all nodes except node 7 7 7 . Notice that relations ( 37 ) and ( 64 ) hold up to rounding errors.

	Node $x$	$1$	$2$	$3$	$4$	$5$	$6$	$7$	Total
exponential	${\overline{\theta }}^{\prime }$	$2.37$	$3.30$	$1.74$	$2.62$	$0.25$	$0.61$	$0.00$	$10.90$
	$\tilde{\theta }$	$0.60$	$0.40$	$0.85$	$0.77$	$1.30$	$1.29$	$1.00$	$5.21$
	$\theta$	$2.97$	$3.70$	$2.58$	$3.39$	$1.55$	$1.91$	$1.00$	$16.10$
	$\kappa$	$0.45$	$0.67$	$0.83$	$0.67$	$0.50$	$0.59$	$-$	$-$
	$\overline{T}$	$1.08$	$2.20$	$1.45$	$1.75$	$0.13$	$0.36$	$0.00$	$6.96$
	$\tilde{T}$	$0.27$	$0.27$	$0.71$	$0.51$	$0.65$	$0.76$	$0.00$	$3.17$
	$T$	$1.35$	$2.47$	$2.15$	$2.26$	$0.78$	$1.12$	$0.00$	$10.13$
power law	${\overline{\theta }}^{\prime }$	$1.26$	$1.73$	$0.79$	$1.39$	$0.17$	$0.42$	$0.00$	$5.77$
	$\tilde{\theta }$	$0.51$	$0.49$	$0.84$	$0.62$	$1.07$	$1.07$	$1.00$	$4.60$
	$\theta$	$1.77$	$2.23$	$1.64$	$2.01$	$1.24$	$1.48$	$1.00$	$10.36$
	$\kappa$	$0.24$	$0.40$	$0.45$	$0.40$	$0.33$	$0.27$	$-$	$-$
	$\overline{T}$	$0.30$	$0.69$	$0.36$	$0.55$	$0.06$	$0.11$	$0.00$	$2.08$
	$\tilde{T}$	$0.12$	$0.20$	$0.38$	$0.25$	$0.36$	$0.29$	$0.00$	$1.59$
	$T$	$0.42$	$0.89$	$0.74$	$0.80$	$0.41$	$0.40$	$0.00$	$3.67$
Weilbull	${\overline{\theta }}^{\prime }$	$5.47$	$6.91$	$4.14$	$5.23$	$0.25$	$0.58$	$0.00$	$22.59$
	$\tilde{\theta }$	$0.75$	$0.25$	$0.87$	$0.89$	$1.61$	$1.58$	$1.00$	$5.95$
	$\theta$	$6.22$	$7.16$	$5.01$	$6.13$	$1.87$	$2.16$	$1.00$	$28.54$
	$\kappa$	$0.67$	$0.79$	$0.87$	$0.79$	$0.63$	$0.78$	$-$	$-$
	$\overline{T}$	$3.68$	$5.48$	$3.60$	$4.15$	$0.16$	$0.46$	$0.00$	$17.51$
	$\tilde{T}$	$0.50$	$0.20$	$0.76$	$0.71$	$1.01$	$1.23$	$0.00$	$4.41$
	$T$	$4.18$	$5.67$	$4.35$	$4.86$	$1.17$	$1.68$	$0.00$	$21.92$

Table 3. Table 3: The mean path lengths of the nonreactive segments ( L ¯ ¯ 𝐿 \bar{L} ), the reactive segments ( L ~ ~ 𝐿 \widetilde{L} ) and the whole first passage path ( L 𝐿 L ) for the maze, the scale-free network, as well as the football match examples.

	$\overline{L}$	$\tilde{L}$	$L$
Maze	$69704.7$	$14129.3$	$83834.0$
Network	$18.01$	$36.83$	$54.84$
Football	$5.19$	$3.47$	$8.65$

Table 4. Table 4: Example 3 3 3 . Various statistics related to the first passage paths of the random work from node 49 49 49 to node 0 0 . The first and the last five nodes of the node lists which are sorted according to values of each statistical quantity are shown. The network is displayed in Figure  8 . Row with label “Degree” shows the degrees of the corresponding nodes. Definitions of other quantities can be found in Section  2 and Table  1 .

Node	$49$	$48$	$14$	$10$	$22$	$\cdots$	$2$	$11$	$7$	$3$	$1$
Degree	$2$	$2$	$2$	$2$	$2$	$\cdots$	$9$	$9$	$11$	$14$	$15$
Node	$49$	$13$	$47$	$11$	$41$	$\cdots$	$8$	$19$	$14$	$48$	$0$
$q$	$0.0$	$0.51$	$0.57$	$0.59$	$0.63$	$\cdots$	$0.74$	$0.75$	$0.86$	$0.87$	$1.0$
Node	$0$	$48$	$14$	$10$	$22$	$\cdots$	$7$	$13$	$3$	$1$	$11$
${\overline{\theta }}^{\prime }$	$0.00$	$0.03$	$0.04$	$0.13$	$0.14$	$\cdots$	$0.83$	$1.09$	$1.14$	$1.18$	$1.40$
Node	$48$	$14$	$10$	$22$	$37$	$\cdots$	$2$	$11$	$7$	$3$	$1$
$\tilde{\theta }$	$0.20$	$0.22$	$0.36$	$0.36$	$0.36$	$\cdots$	$1.63$	$1.99$	$2.05$	$2.67$	$2.83$
Node	$48$	$14$	$10$	$22$	$37$	$\cdots$	$2$	$7$	$11$	$3$	$1$
$\theta$	$0.23$	$0.26$	$0.48$	$0.50$	$0.50$	$\cdots$	$2.24$	$2.88$	$3.39$	$3.81$	$4.01$

Table 5. Table 5: German national team players in the FIFA world cup 2014 2014 2014 final. Information of the substitutions are indicated in the brackets. Notice that player No. 17 17 17 (Mertesacker) does not appear in the graph because he did not contribute to the passes in the trajectories.

No.	Name	No.	Name
$1$	Neuer (GK)	$16$	Lahm (C)
$4$	Höwedes	$18$	Kroos
$5$	Hummels	$20$	Boateng
$7$	Schweinsteiger	$23$	Kramer ($\downarrow {31}^{\prime }$)
$8$	Özil ($\downarrow {120}^{\prime }$)	$9$	Schürrle ($\uparrow {31}^{\prime }$)
$11$	Klose ($\downarrow {88}^{\prime }$)	$17$	Mertesacker ($\uparrow {120}^{\prime }$)
$13$	Müller	$19$	Götze ($\uparrow {88}^{\prime }$)

Table 6. Table 6: Example 4 4 4 . Various statistics related to the first passage paths in the football match. Definitions of these quantities can be found in Section  2 and Table  1 . See Figure  9 and Figure  10 for a depiction of the network.

Node	$q$	${\overline{\theta }}^{\prime }$	$\tilde{\theta }$	$\theta$	Node	$q$	${\overline{\theta }}^{\prime }$	$\tilde{\theta }$	$\theta$
$1$	$0.00$	$0.25$	$0.10$	$0.35$	$16$	$0.40$	$0.64$	$0.43$	$1.07$
$4$	$0.00$	$0.34$	$0.29$	$0.63$	$18$	$0.41$	$0.62$	$0.43$	$1.05$
$5$	$0.00$	$0.39$	$0.14$	$0.53$	$20$	$0.00$	$0.46$	$0.27$	$0.73$
$7$	$0.36$	$0.63$	$0.36$	$0.99$	$23$	$0.48$	$0.05$	$0.05$	$0.09$
$8$	$0.47$	$0.41$	$0.36$	$0.77$	$9$	$0.45$	$0.34$	$0.28$	$0.62$
$11$	$0.68$	$0.07$	$0.15$	$0.22$	$19$	$0.47$	$0.08$	$0.08$	$0.16$
$13$	$0.55$	$0.28$	$0.35$	$0.63$	B$0$	$1.00$	$0.00$	$0.24$	$0.24$
					B$1$	$1.00$	$0.00$	$0.06$	$0.06$

Equations

$$\displaystyle\tau_{A,x}=\min_{k\geq 0}\big{\{}\,k~{}\big{|}~{}x_{0}=x\,,x_{k}\in A\big{\}}\,,\quad\tau_{B,x}=\min_{k\geq 0}\big{\{}\,k~{}\big{|}~{}x_{0}=x\,,x_{k}\in B\big{\}}\,,$$

$$\displaystyle\sigma_{x}=\max_{l\geq 0}\Big{\{}l~{}\Big{|}~{}x_{l}\in A,\,0\leq l<\tau_{B,x}\Big{\}}\,,$$

$$\displaystyle\Xi_{non}=\Big{\{}(x_{0},x_{1},\cdots,x_{\sigma})\Big{\}}\,,\qquad\Xi_{r}=\Big{\{}(x_{\sigma},x_{\sigma+1},\cdots,x_{\tau_{B}})\Big{\}}\,,$$

$$q(x)=\mathbb{P}({\tau }_{B,x}<{\tau }_{A,x}),\forall x\in V,$$

$$\begin{array}{rl}& \sum _{y\in V}p(y|x)q(y)=q(x),\forall x\in (A\cup B{)}^c,\\ & q{|}_A=0,q{|}_B=1.\end{array}$$

$$\displaystyle V^{-}=\Big{\{}x\,\Big{|}\,q(x)<1,\,x\in V\Big{\}},\qquad V^{+}=\Big{\{}\,x~{}\Big{|}~{}\sum_{z\in V}p(z\,|\,x)q(z)>0\,,x\in B^{c}\Big{\}}\,.$$

$$\displaystyle\bar{p}(x\,|\,y)=\left\{\begin{array}[]{cl}\frac{p(x\,|\,y)(1-q(x))}{1-q(y)}\,,&\mbox{if}~{}x\neq 0\,,\\ \sum\limits_{z\in V}p(z\,|\,y)\,q(z)\,,&\mbox{if}~{}x=0\,,~{}y\in A\,,\\ 0\,,&\mbox{if}~{}x=0\,,~{}y\not\in A\,,\\ \end{array}\right.$$

$$\displaystyle\mathbb{P}\big{(}x_{k+1}=x\,\big{|}\,(x_{k}=y,\cdots,x_{0}),\mbox{being nonreactive}\big{)}$$

$$=$$

$$\bar{\theta }(x)=⟨\sum _{l=0}^{\sigma }{1}_x(x_l)⟩,x\in V^{-},$$

$$\sum _{x\in V^{-}}\bar{\theta }(x)=⟨\sum _{l=0}^{\sigma }\sum _{x\in V^{-}}{1}_x(x_l)⟩=⟨\sigma ⟩+1.$$

$$\bar{\theta }(x)=\mu (x)1_A(x)+\sum _{y\in V^{-}}\bar{\theta }(y)\bar{p}(x|y),$$

$$\bar{p}(x|y)=\frac{⟨\sum _{l=0}^{\sigma }{1}_y(x_l)1_x(x_{l+1})⟩}{⟨\sum _{l=0}^{\sigma }{1}_y(x_l)⟩}$$

$$\sum _{y\in V^{-}}\bar{\theta }(y)\bar{p}(x|y)=\sum _{y\in V^{-}}⟨\sum _{l=0}^{\sigma }{1}_y(x_l)1_x(x_{l+1})⟩=⟨\sum _{l=1}^{\sigma +1}{1}_x(x_l)⟩.$$

$$\sum _{y\in V^{-}}\bar{\theta }(y)\bar{p}(x|y)=⟨\sum _{l=0}^{\sigma }{1}_x(x_l)⟩-⟨1_x(x_0)⟩=\bar{\theta }(x)-\mu (x)1_A(x),$$

$$\displaystyle\sum_{x\in A}\bar{\theta}(x)\Big{[}\sum_{z\in V}p(z\,|\,x)q(z)\Big{]}=\sum_{x\in A}\bar{\theta}(x)\bar{p}(0\,|\,x)=1\,.$$

$${\mu }_r(x)=⟨1_x(x_{\sigma })⟩,x\in A,$$

$${\bar{\theta }}^{\prime }(x)=⟨\sum _{l=0}^{\sigma -1}{1}_x(x_l)⟩=\bar{\theta }(x)-{\mu }_r(x)1_A(x),x\in V^{-},$$

$$\displaystyle\begin{split}\mu_{r}(x)=&\bar{\theta}(x)\bar{p}(0\,|\,x)=\bar{\theta}(x)\Big{[}\sum_{z\in V}p(z\,|\,x)q(z)\Big{]}\,,\\ \bar{\theta}^{\prime}(x)=&\bar{\theta}(x)-\mu_{r}(x)=\bar{\theta}(x)\sum_{z\in V^{-}}\bar{p}(z\,|\,x)\,.\end{split}$$

$${\mu }_r(x)=⟨1_x(x_{\sigma })⟩=⟨\sum _{l=0}^{\sigma }{1}_x(x_l)1_0(x_{l+1})⟩,$$

$$\bar{p}(0|x)=\frac{⟨\sum _{l=0}^{\sigma }{1}_x(x_l)1_0(x_{l+1})⟩}{⟨\sum _{l=0}^{\sigma }{1}_x(x_l)⟩}=\frac{{\mu }_r(x)}{\bar{\theta }(x)},x\in A.$$

$$\displaystyle\mu_{r}(x)=\bar{\theta}(x)\bar{p}(0\,|\,x)=\bar{\theta}(x)\Big{[}\sum_{z\in V}p(z\,|\,x)q(z)\Big{]}\,.$$

$$\bar{J}(x\to y)=\bar{\theta }(x)\bar{p}(y|x),x,y\in V^{-},$$

$$\bar{J}(x\to y)=⟨\sum _{l=0}^{\sigma }{1}_x(x_l)1_y(x_{l+1})⟩,$$

$$\displaystyle\begin{split}&\sum\limits_{x\in V^{-}}\bar{J}(x\rightarrow y)=\bar{\theta}(y)-\mu(y)\mathbf{1}_{A}(y)\,,\qquad\forall y\in V^{-}\,,\\ &\sum\limits_{y\in V^{-}}\bar{J}(x\rightarrow y)=\bar{\theta}(x)\Big{[}1-\mathbf{1}_{A}(x)\sum_{z\in V}p(z\,|\,x)q(z)\Big{]}=\bar{\theta}^{\prime}(x)\,,\qquad\forall x\in V^{-}\,.\end{split}$$

$$\displaystyle\widetilde{p}(x\,|\,y)=\left\{\begin{array}[]{ll}\frac{p(x\,|\,y)q(x)}{q(y)}\,,&\mbox{if}~{}y\in V^{+}\cap A^{c}\,,\\ \frac{p(x\,|\,y)\,q(x)}{\sum\limits_{z\in V}p(z\,|\,y)\,q(z)}\,,&\mbox{if}~{}y\in V^{+}\cap A\,,\\ \delta_{y}(x)\,,&\mbox{if}~{}y\in B\,,\end{array}\right.$$

$$\tilde{\theta }(x)=⟨\sum _{l=\sigma }^{{\tau }_B}{1}_x(x_l)⟩,$$

$$\sum _{x\in V}\tilde{\theta }(x)=⟨{\tau }_B⟩-⟨\sigma ⟩+1.$$

$$\tilde{\theta }(x)=\sum _{y\in V^{+}}\tilde{\theta }(y)\tilde{p}(x|y),\forall x\in A^c.$$

$$\tilde{\theta }(x)=⟨1_x(x_{{\tau }_B})⟩,x\in B.$$

Full text

1 Introduction

(Markov) jump process has been extensively studied in the past decades and nowadays it becomes a standard mathematical model that is widely applied to problems arising from physics, chemistry, biology, etc [7, 24]. Among many important topics in the study of jump processes, the first passage paths have attracted considerable attention within different disciplines, where the main purpose is to understand the transition mechanism of the system between different subsets in state space and to e.e. access how much time the transitions typically take [21]. Examples include reaction networks in chemistry [38, 37], phenotypic switches in cell biology [39], conformational changes in molecular dynamics [17, 25], disease spreading within certain geographical areas [4, 6], as well as spread of information on social networks [16]. In these contexts, studies of the first passage paths are often very helpful to understand the underlying processes and to foster- or prevent the transition events.

A common situation that one often encounters in the study of many real-world applications is that the system exhibits metastability and transition events become very rare [27, 15, 30]. To study the transition events in these scenarios, transition path theory (TPT) has been developed both for diffusion processes on continuous state space [9, 8, 34] and for Markov jump processes on discrete state space [23, 5]. It provides a probabilistic framework to analyze the statistical properties of system’s reactive trajectories (following the terminology in [19]) and enables us to answer the questions which are important in order to understand the transition mechanism of the system. In the discrete state space setting, TPT can be applied to study Markov jump processes which are ergodic with a unique invariant measure [23, 35].

Meanwhile, inspired by the wide applications that have emerged due to the rapid development of network science, scientific interest has been extended to study processes which go beyond ergodic Markov jump processes [26, 29, 28]. One simple example when ergodicity is violated is the random walk on a directed graph which contains sink-states (states with no outward edges). Another example of a non-Markovian process is the continuous-time random walk with possibly non-exponential waiting time distributions. These kind of processes have been applied to model the burst and memory effects in real-world networks [13, 20, 10]. The above described processes are no longer ergodic Markov jump processes and therefore TPT can not be applied directly. However, interesting insights can be obtained when studying the transition mechanism from one subset to another, to determine how much time the transitions typically take and to identify key nodes or edges for the transition events. To our best knowledge, these questions have not been systematically addressed in the non-Markovian and non-ergodic setting (see related study in [20]).

In the current work we consider the first passage path ensemble of both continuous-time and discrete-time jump processes on a discrete, finite state space. While we are strongly influenced by TPT and will study similar quantities such as the probability of visiting each node and probability fluxes on each edge, we emphasize that both the subject being studied and the setting are different from the study of TPT [23, 5]. Specifically, we will study the first passage path by dividing it into nonreactive and reactive segments. While in the ergodic case the reactive segments coincide with the reactive trajectories in TPT, in this work we also study the statistics of nonreactive segments and their relations with the statistics of the entire first passage paths. Furthermore, in our study the processes are allowed to be non-ergodic, and in the continuous-time scenario the waiting time on each node can be also non-exponentially distributed. The main contributions of this paper can be summarized as follows. Firstly, statistical properties of both nonreactive and reactive path ensembles have been analyzed, which enables us to compute several important quantities associated to the reactive- and nonreactive ensembles. Their relations to the entire first passage path ensemble are obtained subsequently. Secondly, since the processes are not necessarily ergodic or in stationary anymore, our analysis (of the reactive segments) can be viewed as an extension of TPT to nonequilibrium path ensembles. When further assuming that the processes are ergodic Markov jump processes and the first passage path ensemble is in equilibrium, our analysis recovers TPT and allows to elucidate the statistical connection between transition paths in TPT and the first passage paths. Thirdly, the numerical implementation of the derived results is discussed, which is useful in many applications.

The paper is organized as follows: The first passage path ensemble of discrete-time Markov jump processes is studied in Section 2. The first passage path ensemble of continuous-time jump processes with general waiting time distributions is considered in Section 3. Connections of our analysis with TPT are discussed in Section 4 where we recover the results of TPT when the processes are ergodic and the path ensemble is in equilibrium. Algorithmic issues are discussed in Section 5 and several numerical examples are presented in Section 6 to illustrate our analysis framework. Conclusions and further discussions are present in Section 7. Finally, some supplementary analysis related to Section 2 and Section 3 is provided in Appendix A and Appendix B.

For the reader’s convenience, important notations which will be used in the current work are summarized in Table 1.

2 Analysis of the first passage path ensemble : discrete-time Markov jump processes

In this section, we study the first passage path ensemble when the system is a discrete-time Markov jump process. After introducing various useful notations in Subsection 2.1, we will analyze the statistics of the nonreactive ensemble and the reactive ensemble in Subsection 2.2 and Subsection 2.3, respectively. Their connections with the whole first passage path ensemble are discussed in Subsection 2.4.

We also emphasize that although several quantities that we will consider are strongly influenced by TPT, the main difference is that they are related to the first passage path ensemble which may be out of equilibrium and therefore do not reply on the existence of invariant measure. Connections of our analysis with TPT will be further discussed in Section 4.

2.1 Preparations

We start by introducing the processes we will consider and the notations that will be used in this paper. Let $G=(V,E)$ be the graph representation of a directed network $G$, where $V$ is the set of nodes and $E\subseteq V\times V$ is the set of edges. We assume that $V$ is a finite set and, without loss of generality, $V=\{1,2,\cdots,n\}$ for some $n>1$. We will write $x\rightarrow y$ if $(x,y)\in E$ and denote $\mathcal{N}_{x}=\{\,y\in V~{}|~{}x\rightarrow y\}$ as the set consisting of all nodes which can be directly reached from node $x$. For simplicity, we assume that there are no loop edges in $G$, i.e. $x\not\in\mathcal{N}_{x}$, for $\forall x\in V$. Since we are also interested in the case when the network $G$ contains sinks, we allow the case when $\mathcal{N}_{x}=\emptyset$ for some node $x\in V$ and denote $\mathcal{T}=\{x\in V~{}|~{}\mathcal{N}_{x}=\emptyset\}$ to be the set of sink nodes.

Suppose that two disjoint nonempty subsets $A,B\subset V$ are given. And let $\mu$ be a probability distribution on set $A$. Consider the discrete-time Markov process defined by the jump probability $p(y\,|\,x)$ for $x,y\in V$, which is the probability that the system will jump to state $y$ if its current state is $x$. Given node $x\in V$, we define two stopping times related to sets $A,B$

[TABLE]

and set $\tau_{A,x}=+\infty$ (or $\tau_{B,x}=+\infty$) if the process will never reach set $A$ (or $B$) from $x$. Especially, we have $\tau_{A,x}\equiv 0$ for $x\in A$ and $\tau_{B,x}\equiv 0$ for $x\in B$. Assume the process starts from some node $x\in B^{c}$ and consider the path until it reaches set $B$. Denote the path as $(x_{0},x_{1},\cdots,x_{k})$, then we have $x_{0}=x$, $x_{k}\in B$, where $k=\tau_{B,x}$ and $x_{l}\not\in B$ for $0\leq l<k$. Such a path is called the first passage path [6, 21]. We also define the last hitting time of set $A$ as

[TABLE]

and set $\sigma_{x}=+\infty$, if $x_{l}\not\in A$ for $0\leq l<\tau_{B,x}$. In the following, we will use the notations $\tau_{A}$, $\tau_{B}$ and $\sigma$ for simplicity if we do not explicitly emphasize the initial state $x$.

Given any first passage path starting from a state in set $A$ (therefore $\sigma<+\infty$), we can split this path into two segments using the last hitting time $0\leq\sigma<\tau_{B}$. The first segment $(x_{0},x_{1},\cdots,x_{\sigma})$, which will be termed as nonreactive trajectory or nonreactive segment, consists of the consecutive nodes visited by the process before it eventually leaves set $A$. The second segment $(x_{\sigma},x_{\sigma+1},\cdots,x_{\tau_{B}})$, which is called reactive trajectory in transition path theory [23], is the transition pathway of the process from set $A$ to set $B$ (see Figure 1(a) for illustration). We will call it either reactive trajectory or reactive segment. Equivalently, the reactive trajectory is the segment of the first passage path starting from some node in $A$ which is hit by the process for the last time. Clearly, we have $x_{\sigma}\in A$ and $x_{l}\in(A\cup B)^{c}$ for $\sigma<l<\tau_{B}$. The ensembles of the nonreactive trajectories and reactive trajectories are denoted by

[TABLE]

respectively, where $(x_{0},x_{1},\cdots,x_{\tau_{B}})$ goes over all first passage paths with initial state $x_{0}\in A$ and $x_{0}\sim\mu$.

Given a node $x\in V$, we say that set $A$ (or $B$) is reachable from $x$, if there is a path of finite length $x_{0},x_{1},\cdots,x_{k}$, where $k\geq 0$, such that $x_{0}=x$, $x_{k}\in A$ (or $x_{k}\in B$), $x_{l}\in(A\cup B)^{c}$, $1\leq l\leq k-1$, and $p(x_{l+1}\,|\,x_{l})>0$, $0\leq l\leq k-1$. Throughout the paper, we will make the following assumption.

Assumption 1**.**

$(A\cup B)^{c}\not=\emptyset$. For each node $x\in(A\cup B)^{c}$, either set $A$ or set $B$ is reachable from $x$. Furthermore, set $B$ is reachable from each node $x\in A$.

Under the above assumption, we can conclude by contradiction that $\tau_{A,x}\wedge\tau_{B,x}<+\infty$ with probability $1$ for all $x\in V$. Therefore, the committor function,

[TABLE]

corresponding to sets $A,B$ is well defined and will play an important role in the following study. It is known that it satisfies the equations [23]

[TABLE]

We also define two subsets

[TABLE]

Clearly, we have $A\subseteq V^{-}\subseteq B^{c}$ and $A\subseteq V^{+}\subseteq B^{c}$, where the latter is implied by Assumption 1 together with the following result.

Proposition 1**.**

Let $\sigma_{x}$ be the last hitting time defined in (2) and subsets $V^{-},V^{+}$ be defined in (6). We have

$\mathbb{P}(\sigma_{x}<+\infty)=1-q(x)$, $\forall x\in V$. 2. 2.

$x\in V^{-}$* iff $x\in B^{c}$ and set $A$ is reachable from node $x$.* 3. 3.

$x\in V^{+}$* iff $x\in B^{c}$ and set $B$ is reachable from node $x$.*

Proof.

We will only prove the first two conclusions since the third one can be obtained using a similar argument as the second one.

By definition of $\sigma_{x}$, we know that the two events $\sigma_{x}<+\infty$ and $\tau_{A,x}<\tau_{B,x}$ are equivalent. It follows from the definition of $q$ in (4) that $\mathbb{P}(\sigma_{x}<+\infty)=\mathbb{P}(\tau_{A,x}<\tau_{B,x})=1-q(x)$. 2. 2.

Suppose $x\in V^{-}$. Using the definition of $q$, we know $\tau_{A,x}<+\infty$ with a positive probability, which implies that set $A$ is reachable from $x$. Conversely, suppose $x\in B^{c}$ and set $A$ is reachable along the path $x_{0}=x,x_{1},\cdots,x_{k}\in A$. If $x\not\in V^{-}$, then $q(x)=1$ and $x\in(A\cup B)^{c}$. Applying equation (5), we obtain $q(x_{1})=1$. As $x_{l}\in(A\cup B)^{c}$ for $1\leq l\leq k-1$, we can repeat the argument and obtain $q(x_{k})=q(x_{k-1})=\cdots=q(x_{1})=1$. However, this contradicts the fact that $q|_{A}=0$ since $x_{k}\in A$. Therefore we have $x\in V^{-}$.

∎

From Proposition 1, we know that Assumption 1 implies $V^{-}\cup V^{+}=B^{c}$ and $\mathcal{T}\subset B$.

2.2 Nonreactive ensemble

In this subsection, we study the nonreactive ensemble $\Xi_{non}$ defined in (3). Given a nonreactive trajectory $x_{0},x_{1},\cdots,x_{\sigma}$, where $x_{0}\in A$ and $\sigma$ is the last hitting time defined in (2), it is clear that $x_{l}\in V^{-}$ for $0\leq l\leq\sigma$ (Proposition 1). Our aim is to define a Markov jump process whose path ensemble coincides with $\Xi_{non}$. For this purpose, we first introduce a virtual node [math] to mark the end of the path and consider the extended node set $V^{-}\cup\{0\}$. A jump from some node $x\in A$ to node [math] indicates that $x$ is the last node of the nonreactive trajectory. We also define the transition probability from node $y\in V^{-}$ to $x\in V^{-}\cup\{0\}$ as

[TABLE]

and $\bar{p}(x\,|\,0)=\delta(x)$. Using (5), we can verify that $\bar{p}$ is a probability matrix on set $V^{-}\cup\{0\}$ with row sum one. Furthermore, we have

Proposition 2**.**

The ensemble $\Xi_{non}$ can be generated by trajectories (without the end node [math]) of the Markov jump process on space $V^{-}\cup\{0\}$ with transition probability matrix $\bar{p}$ in (10) and initial distribution $\mu$.

Proof.

Let $(x_{0},x_{1},\cdots,x_{k},\cdots,x_{\sigma})\in\Xi_{non}$ be one nonreactive trajectory such that $x_{k}=y$. For $y\in V^{-}\cap A^{c}$ and $x\in V^{-}$, using the Markov property and the definition of the committor function $q$, we can compute

[TABLE]

For $y\in A$, event $\{k=\sigma\}$ is equivalent to the event that the original process (on $V$) hits set $B$ first (before it returns to $A$) after it leaves set $A$ from node $y$. The probability of the latter is equal to $\sum\limits_{z\in V}p(z\,|\,y)q(z)$ and therefore coincides with the probability that the process (on $V^{-}\cup\{0\}$) jumps from $y$ to [math]. The case for $x\in V^{-},y\in A$ can be argued using a similar argument. ∎

Given $x\in V^{-}$, let $\bar{\theta}(x)$ be the average number of times that node $x$ is visited by the nonreactive trajectories, i.e.

[TABLE]

where $\mathbf{1}_{x}(\cdot)$ is the indicator function, $\langle\cdot\rangle$ denotes the ensemble average of $\Xi_{non}$, or equivalently, the ensemble average of the first passage paths with the initial distribution $\mu$. It is straightforward to verify

[TABLE]

Furthermore, we have

Proposition 3**.**

For $x\in V^{-}$, function $\bar{\theta}(\cdot)$ defined in (11) satisfies the equation

[TABLE]

where probabilities $\bar{p}$ are given in (10).

Proof.

Using Proposition 2, we can rewrite the transition probabilities $\bar{p}$ using the ensemble average, i.e.

[TABLE]

where $x,y\in V^{-}$ and we have set $x_{\sigma+1}=0$, which marks the end of the nonreactive trajectory. Therefore, using the definition of $\bar{\theta}$ in (11) and summing up $y\in V^{-}$, we obtain

[TABLE]

Since $x\in V^{-}$, we know $\mathbf{1}_{x}(x_{\sigma+1})=0$ and therefore

[TABLE]

where we have used the definition of $\bar{\theta}$ again and the fact that $x_{0}\sim\mu$. ∎

Especially, summing up $x\in V^{-}$ in (13) and using the fact that $\bar{p}$ has row sum one, we can verify the equality

[TABLE]

It is also useful to introduce

[TABLE]

which is the probability distribution of the last hitting node on set $A$ and satisfies $\sum\limits_{x\in A}\mu_{r}(x)=1$ (see Figure 1 (b) for illustration). Furthermore, in analogy to (11), we define

[TABLE]

which coincides with $\bar{\theta}$ on $A^{c}$ and the summation above is interpreted to be zero when $\sigma=0$. The following relations are straightforward.

Proposition 4**.**

Let $\bar{\theta}$ and the probabilities $\bar{p}$ be defined in (11) and (10), respectively. Then for $x\in A$,

[TABLE]

Proof.

We only need to prove the first equality in (19). Notice that (17) can be written as

[TABLE]

where we have used the convention that $x_{\sigma+1}=0$. Similarly as in (14), applying Proposition 2, we have

[TABLE]

Therefore, we conclude that

[TABLE]

∎

We also introduce the nonreactive probability flux, which is defined as

[TABLE]

and equals zero for other edges in $E$. From (11) and (14), $\bar{J}$ can be written as a path ensemble average

[TABLE]

i.e. the average number of times that edge $x\rightarrow y$ is visited by the nonreactive trajectories. Applying Proposition 3 and Proposition 4, we have

Corollary 1**.**

[TABLE]

2.3 Reactive ensemble

In this subsection, we turn to study the reactive ensemble $\Xi_{r}$ in (3). First of all, from the definition (3), we know that the probability distribution of the initial state $x_{\sigma}$ of the ensemble $\Xi_{r}$ coincides with the probability distribution $\mu_{r}$ of the end node in the ensemble $\Xi_{non}$. See (17) and Proposition 4 (an alternative derivation can be found in Appendix A).

Now recall the definition of set $V^{+}$ in (6), Proposition 1, and also notice that $A\subseteq V^{+}$. In analogy to the previous subsection, our aim is to construct a Markov jump process on space $V^{+}\cup B$ whose path ensemble coincides with $\Xi_{r}$. To do so, we define the transition probabilities

[TABLE]

where $x,y\in V^{+}\cup B$, and $\delta_{y}$ is the delta function centered at $y$. We have the following result (the proof is omitted since it is similar to Proposition 2; Note that a similar result under the assumption of ergodicity has been obtained in [5]).

Proposition 5**.**

The ensemble $\Xi_{r}$ can be generated from the trajectories of the Markov jump process on space $V^{+}\cup B$ which is defined by the transition probabilities $\widetilde{p}$ in (26) and the initial distribution $\mu_{r}$ on $A$.

For $x\in V$, let $\widetilde{\theta}(x)$ be the average number of times that node $x$ has been visited by reactive trajectories, i.e.

[TABLE]

where $(x_{0},x_{1},\cdots,x_{\sigma},\cdots,x_{\tau_{B}})$ is a first passage path and $\langle\cdot\rangle$ denotes the corresponding ensemble average. Clearly $\widetilde{\theta}(x)=0$ when $x\not\in V^{+}\cup B$. Summing up $x\in V$ in (27), we can obtain

[TABLE]

Similar to Proposition 3, we have

Proposition 6**.**

Let $\widetilde{p}$ be the transition probability defined in (26). We have $\widetilde{\theta}(x)=\mu_{r}(x)$ for $x\in A$, and

[TABLE]

Especially, $\sum\limits_{x\in B}\widetilde{\theta}(x)=1$.

Proof.

Since nodes of set $A$ appear in reactive trajectories only as the starting state $x_{\sigma}$ with probability distribution $\mu_{r}$, we have $\widetilde{\theta}=\mu_{r}$ on $A$. Equation (29) can be proved in an analogous way to Proposition 3, by rewriting $\widetilde{p}$ as the ensemble average and applying Proposition 5. See (14). By the definition of the stopping time $\tau_{B}$, we know $x_{l}\not\in B$ for $l<\tau_{B}$. Therefore, from (27),

[TABLE]

Summing up $x\in B$ and using the fact that $x_{\tau_{B}}\in B$, we conclude

[TABLE]

∎

Remark 1**.**

Notice that (29) holds for node $x\in B$ as well. For numerical computation, we can first compute $\widetilde{\theta}(x)$ for $x\in V^{+}\subseteq B^{c}$ by solving the linear system (29), and then obtain the end node distribution $\widetilde{\theta}(x)$ for each $x\in B$ from (29).

For each edge $x\rightarrow y$, $x\in V^{+}$, $y\in V^{+}\cup B$, we define the reactive probability flux as

[TABLE]

and set it to zero for other edges in $E$. In analogy to (21), we have the ensemble average representation

[TABLE]

where $(x_{0},x_{1},\cdots,x_{\sigma},\cdots,x_{\tau_{B}})$ is a first passage path and $\langle\cdot\rangle$ denotes the corresponding ensemble average. Applying Proposition 6, we can obtain

Corollary 2**.**

Let $\widetilde{J}$ be the reactive probability flux defined in (30) and $\widetilde{\theta}$ be given in (27). We have

[TABLE]

Proof.

It follows directly by applying (30), Proposition 6, and the fact that row sums of matrix $\widetilde{p}$ are one. ∎

2.4 First passage path ensemble

Based on the previous analysis of the nonreactive and reactive trajectories, we study the whole first passage path ensemble in this subsection.

Given $x\in V$, we define $f(x)$ as the mean first hitting time of set $B$ for the discrete-time Markov jump process starting from $x$, i.e. $f(x)=\langle\tau_{B,x}\rangle$. Using Markovianity, we can write it as

[TABLE]

It is also well known that $f$ satisfies the equations

[TABLE]

Accordingly, if the probability distribution of the initial state on set $A$ is $\mu$, the average length of the first passage path is then given by $\sum\limits_{x\in A}\mu(x)f(x)$. Also, let $\theta(x)$ be the average number of times that node $x$ has been visited by the first passage path with initial distribution $\mu$, i.e.

[TABLE]

In analogy to Proposition 3 and Proposition 6, we know it satisfies

[TABLE]

Furthermore, taking into account the definitions of $\bar{\theta}$, $\bar{\theta}^{\prime}$ and $\widetilde{\theta}$ in (11), (18) and (27), as well as relations (12), (28), we can verify the following relations

[TABLE]

In fact, the following explicit relations hold.

Proposition 7**.**

For any $x\in V$,

[TABLE]

Proof.

Clearly $\bar{\theta}(x)=\widetilde{\theta}(x)=0$ if $\theta(x)=0$. Hence we only need to consider the case when $\theta(x)>0$. Using the Markov property, we can compute the probability $\mathbb{P}(\tau_{A,x}<\tau_{B,x})$ from the first passage ensemble. Specifically, let $(x_{0},x_{1},\cdots,x_{\tau_{B}})$ be a first passage path. Conditioning on $x_{l}=x$ where $0\leq l\leq\tau_{B}$, the event $\{\tau_{A,x}<\tau_{B,x}\}$ is actually equivalent to $\{l\leq\sigma\}$. Therefore,

[TABLE]

which implies \bar{\theta}(x)=\theta(x)\big{(}1-q(x)\big{)}. The expression of $\widetilde{\theta}(\cdot)$ follows from (37). ∎

We also introduce the probability flux for edge $x\rightarrow y$, which is defined as

[TABLE]

As in (21) and (31), we have

[TABLE]

And the following result is straightforward.

Proposition 8**.**

For $x\in B^{c}$, we have $\theta(x)p(y\,|\,x)=\bar{\theta}(x)\bar{p}(y\,|\,x)+\widetilde{\theta}(x)\widetilde{p}(y\,|\,x)\,$, or equivalently,

[TABLE]

Proof.

(41) follows directly from the ensemble average representations of fluxes in (40), (21), (31). ∎

3 Analysis of the first passage path ensemble : continuous-time jump processes

In this section, we turn to study the first passage path ensemble of continuous-time jump processes with general waiting time distributions. After introducing the process and some notations, we will derive a discrete-time Markov jump process which encodes the dynamical information of the original continuous-time process. Applying the analysis in Section 2 to this discrete-time Markov jump process, we are able to analyze the first passage path ensemble of the continuous-time jump process.

3.1 Preparations

We consider a continuous-time jump process on network $G=(V,E)$. We will follow the derivations in [13] and some extra notations are needed in order to introduce the process. A related study on the continuous-time jump processes can also be found in [20].

Following Subsection 2.1, let us first assume that node $x\not\in\mathcal{T}$ and consequently $\mathcal{N}_{x}\not=\emptyset$. From node $x$, the system may jump to one of the nodes in $\mathcal{N}_{x}$ after staying at $x$ for a certain duration of time. We assume that the jump events at each node $x$ are independent of each other, and denote the probability density of the waiting time $t$ as $\psi(t\,|\,x\rightarrow y)$, conditioning on that the system jumps from $x$ to another node $y\in\mathcal{N}_{x}$. As a probability density function, it satisfies $\int_{0}^{+\infty}\psi(t\,|\,x\rightarrow y)\,dt=1$. Let $a(t\,|\,x)$ be the probability that the system stays at node $x\in V$ for a time longer than $t$ before it leaves. Also denote the probability density that the system jumps from node $x$ to $y$ at time $t$ by $b(t\,,x\rightarrow y)$. Since we have assumed that the jump events to different nodes are independent of each other, it is straightforward to verify that

[TABLE]

and they satisfy the equation

[TABLE]

We also know that the average time the system will stay at a node $x\in V$ is

[TABLE]

Let $p(y\,|\,x)$ be the probability that the system jumps from node $x$ to $y$, regardless of when the jump event occurs, then we have

[TABLE]

and $p(y\,|\,x)=0$, for $y\not\in\mathcal{N}_{x}$. From (42) it follows $\lim\limits_{t\rightarrow+\infty}a(t\,|\,x)=0$ and, taking the limit $t\rightarrow+\infty$ in (43), we can obtain

[TABLE]

We also assume that the process will terminate once it reaches one of the sink nodes. Correspondingly, for $x\in\mathcal{T}$, we set $p(y\,|\,x)=\delta_{x}(y)$, $y\in V$, and therefore the equality $\sum_{y\in V}p(y\,|\,x)=1$ is still satisfied. In other words, $p$ is a probability matrix and therefore defines a discrete-time Markov jump process on $G$. As further presented in Appendix B, this discrete-time Markov jump process encodes the dynamical information of the original continuous-time process. And it will be useful in the following Subsection 3.2.

For general probability densities $\psi$, numerical integrations are needed in order to compute transition probabilities $p(y\,|\,x)$ from (42) and (45). In the following, we discuss three special cases [10, 32] when the analytical expressions of $p(y\,|\,x)$ can be obtained (see Figure 2).

Exponential distributions. The probability densities are

[TABLE]

for some $\lambda_{xy}>0$, where $y\in\mathcal{N}_{x}$. It corresponds to the continuous-time Markov jump process and we can define the generator matrix $L$ of the process whose entries are

[TABLE]

for $x\not\in\mathcal{T}$, and $L_{xy}=0$, $\forall y\in V$, when $x\in\mathcal{T}$. From (42), (45) we can obtain that, for $x\not\in\mathcal{T}$,

[TABLE] 2. 2.

Weibull distributions. In this case, we assume

[TABLE]

for some $k>0$ and $\lambda_{xy}>0$, where $y\in\mathcal{N}_{x}$. From (42), (45) we can obtain that, for $x\not\in\mathcal{T}$,

[TABLE]

where $\Gamma(z)=\int_{0}^{+\infty}x^{z-1}e^{-x}dx$ is the Gamma function. Clearly, we recover the exponential distribution when $k=1$. 3. 3.

Power law distributions. In this case, we assume

[TABLE]

for some $\alpha_{xy}>1$, where $y\in\mathcal{N}_{x}$. From (42), (45) we can obtain that, for $x\not\in\mathcal{T}$,

[TABLE]

3.2 Path ensemble analysis

In this subsection, we study the path ensemble of the continuous-time jump process by applying the analysis in Section 2 to the discrete-time Markov jump process obtained in the previous Subsection 3.1. The structure of the contents is the same as in Section 2 and therefore only the necessary steps in order to adapt the analysis to the continuous setting will be summarized.

Nonreactive ensemble

For the nonreactive ensemble, denoting $x_{s}$ as the continuous-time jump process introduced in Subsection 3.1 and following Subsection 2.2, we define

[TABLE]

to be the average amount of time that the nonreactive trajectories spend at node $x$. Notice that we denote $t_{l}$ as the time when the $l$th jump occurs and especially $t_{\sigma}$ is the last hitting time of set $A$. Recall (44), we have

[TABLE]

and therefore the average total amount of time of the nonreactive trajectories is

[TABLE]

Reactive ensemble

For the reactive ensemble, similar to (57), we define

[TABLE]

which is the average amount of time that the reactive trajectories spend at node $x$. From (44), we have

[TABLE]

and therefore the average total amount of time of the reactive trajectories is

[TABLE]

First passage path ensemble

For the first passage path ensemble, following Subsection 2.4, we define

[TABLE]

which is the average amount of time that the first passage trajectories spend at node $x$. Similarly as above, we can obtain

[TABLE]

where $\bar{T},\widetilde{T}$ are defined in (57), (60), and the average total amount of time of the first passage trajectories is

[TABLE]

4 Ergodic case : connections with TPT

In this section, we consider the special case when either the original process or, in the continuous-time case, the discrete-time Markov jump process obtained in Section 3 is both irreducible and aperiodic. In this case, statistics of the transition (reactive) segments have been studied in TPT by embedding these segments into an infinitely long stationary trajectory. Our key observation is that in fact they can be embedded into the first passage path ensemble with certain initial distribution $\mu$ and therefore can be studied by applying the analysis in the previous sections. This approach elucidates the relations between the study of the first passage paths and the study of the transition paths in TPT.

To proceed, we first notice that in this case Assumption 1 in Section 2 is satisfied and indeed we have $V^{-}=V^{+}=B^{c}$ in (6), $\mathcal{T}=\emptyset$. The discrete-time jump process is ergodic and we assume its unique invariant measure is $m$ such that $\sum_{x\in V}m(x)=1$ [24, 18]. We also introduce the time reversed process defined by the transition probabilities $p^{-}(y\,|\,x)=\frac{m(y)p(x\,|\,y)}{m(x)}$, $x,y\in V$ and the backward committor function $q^{-}$ which satisfies the equation

[TABLE]

It is known that $q^{-}(x)$ equals to the probability that, being at $x\in V$, the process came from set $A$ rather than from set $B$ [23, 34].

We can further assume that there is an infinitely long trajectory $x_{0},x_{1},\cdots,x_{l},\cdots$ of the discrete-time jump process, where $x_{0}\in V$, $x_{0}\sim m$. To make a connection with the general case studied in the previous subsections, we introduce the stopping times $\tau^{(0)}_{A}=\tau^{(0)}_{B}\equiv 0$, and

[TABLE]

where $k\geq 0$. Similarly as in (2) and (3), we define

[TABLE]

and consider the ensembles of the equilibrium nonreactive and reactive segments

[TABLE]

constructed from the infinitely long trajectory $x_{0},x_{1},\cdots$. Using the definition of $q^{-}$, we can obtain the expressions of the various quantities related to ensembles (69) in the ergodic setting.

Proposition 9**.**

Consider the path ensembles $\Xi_{non}$ and $\Xi_{r}$ in (69).

For the normalization constant, we have

[TABLE] 2. 2.

The initial distribution $\mu$ (on set $A$) of the ensemble $\Xi_{non}$ satisfies

[TABLE] 3. 3.

Let $\theta$ be defined in (35), we have

[TABLE] 4. 4.

Let $\mu_{r}$ be the initial distribution of $\Xi_{r}$, and $\bar{\theta}$, $\widetilde{\theta}$ be defined in (11), (27) respectively. We have

[TABLE]

Proof.

We have

[TABLE]

And it follows from (66) that

[TABLE]

Therefore, using $q^{-}|_{A}=1$, $q^{-}|_{B}=0$, from (77) we obtain

[TABLE]

which implies (70). 2. 2.

Using Markovianity of the discrete-time jump process and the definition of $q^{-}$, we can derive

[TABLE]

where $\propto$ means “equal up to a constant” and the normalization constant is given in (70). 3. 3.

Using the equation of $q^{-}$ in (66) and the formula of $\mu$ in (71), it is easy to verify that $\theta$ defined in (75) satisfies equations (36). 4. 4.

The expressions of $\bar{\theta}$, $\mu_{r}$ and $\widetilde{\theta}$ follows from (75), Proposition 4 and Proposition 7.

∎

Remark 2**.**

Using the expression of $\mu_{r}$ in (76) and an argument similar to the proof of (70), we could obtain

[TABLE]

Finally, we consider the continuous-time jump process introduced in Section 3 and assume its associated discrete-time Markov jump process obtained in Subsection 3.1 is ergodic. In this case, we consider an infinitely long trajectory of the process as in (93) in Appendix B and let $M=M(N)>0$ be the integer such that $\tau_{B}^{(M)}\leq N<\tau_{B}^{(M+1)}$, where the stopping times $\tau_{B}^{(M)}$ are defined in (67). Following [23], we define the reaction rate $k_{AB}$ between sets $A$ and $B$ by

[TABLE]

We have the following result.

Proposition 10**.**

Consider the continuous-time jump process introduced in Subsection 3.1. Let $\kappa$ be defined in (44). $p,q$ are the transition probabilities and the committor function of the associated discrete-time Markov jump process defined in (45), (4), respectively. Suppose that this discrete-time jump process is ergodic with a unique invariant measure $m$. We have

[TABLE]

Proof.

We consider the path in (93) in Appendix B which jumps from state $x_{i}$ to $x_{i+1}$ at time $t_{i+1}$, $i\geq 0$. We also denote its state at time $s\geq 0$ as $x_{s}$. Using the Markov property and ergodicity of the discrete-time jump process, we have

[TABLE]

On the other hand, we have

[TABLE]

Therefore,

[TABLE]

Combining (81) and (82), we can conclude that

[TABLE]

∎

Remark 3**.**

We conclude this section with the following remarks.

From (78) in Remark 2, we know the numerator in the expression (80) equals the constant $Z$ and can be replaced by the other expressions in (78). 2. 2.

The case when the continuous-time jump process itself is Markovian and ergodic has been studied in **[23]**. The invariant measure $\pi$ in **[23]** of the continuous-time process is related to $m$ and $\kappa$ in the present work by

[TABLE]

Together with formulas in (52), we can verify that Proposition 9 and Proposition 10 are accordant with the results in **[23]**. Therefore, we have extended the analysis there to more general continuous-time jump processes introduced in Subsection 3.1.

5 Algorithmic issues

In this section, we briefly discuss some algorithmic issues related to applying the analysis of Section 2 and Sections 3 to applications.

5.1 Summary of the analysis procedure

Given a continuous-time or discrete-time jump process on a finite state space, the analysis of its first passage paths can be proceeded as follows.

Construction of the discrete-time network.

The analysis of the current work can be applied to both continuous-time and discrete-time jump processes. In the case of a continuous-time jump process, we have shown in Section 3 how the transition probabilities $p$ of the corresponding discrete-time Markov jump process (without time information) are related to the probability density function $\psi$ of the waiting times. For certain probability density $\psi$, analytical formulas of $p$, $\kappa$ as well as other quantities in Section 3 can be obtained. For more general probability densities, however, numerical integration is needed in order to compute $p$, $\kappa$ using formulas (42), (45) and (44). 2. 2.

Calculation of the statistics of path ensembles. After obtaining transition probabilities $p$, we can compute various statistical quantities of the nonreactive ensemble, reactive ensemble, as well as the entire first passage path ensemble. While the equations for each quantity have been derived in Section 2 and some quantities can be obtained in several ways, we suggest to proceed in the order which is summarized in Algorithm 1. In the ergodic case where the initial distribution $\mu$ is given in (71), we could either first obtain $q^{-}$ from the backward committor equation (66) and compute the other quantities using Proposition 9, or follow the procedure in [23].

Among the steps in Algorithm 1, the only possible computational difficulties are related to solving the linear systems (5) and (36). Using numerical packages such as PETSc [3], however, these linear systems can be easily solved for (sparse) networks with several thousand nodes and therefore the algorithm could be used in a wide range of applications.

Post-processing. The various statistics obtained above allow us to better understand the mechanism of the system’s transitions from one set to another. However, how to interpret and represent these information is a nontrivial question and indeed depends on the problem at hand. In some cases, the key interest is to find certain important pathways which the system is likely to take [9, 34, 23]. Algorithms for computing dominant pathways in the network setting have been studied in [23, 17, 36]. In general, however, the first passage paths of the system may be typically very long and diffusive. In this case, presenting the statistics by identifying a single representative pathway may be incomplete or even misleading (see examples in Section 6). In the current work, instead of discussing in detail specific methods to further exploit the statistics, we simply point out that one can rank the importance of nodes and edges based on these statistics. Specifically, taking the reactive trajectories as an example,

1. (a)

By ranking nodes according to committor function $q$, we can figure out how “close” each node is to the initial set $A$ and to the terminal set $B$. (Here the closeness is measured by the committor probability). 2. (b)

By ranking nodes according to function $\widetilde{\theta}$, we know which nodes are more frequently visited by the reactive trajectories. 3. (c)

By ranking edges according to flux $\widetilde{J}$, we know which edges are more frequently visited by the reactive trajectories.

In the case of the continuous-time process, ranking can be also based on $\widetilde{T}$ so that we could identify nodes on which the process will spend more time. Certainly, similar rankings allow us to identify nodes and edges which are important for the nonreactive trajectories, as well as for the entire first passage paths.

5.2 Data-based approach

In various real-world applications, we may face the situation that the system’s information is not available, i.e. $p$ or $\phi$ are unknown, and only the trajectories of the system can be observed. In this case, one usually takes a data-based approach and constructs a Markov jump process from the available data. Specifically, suppose that we have obtained $M$ trajectories of the system. For the $i$th trajectory, $1\leq i\leq M$, it starts from $x^{(i)}_{0}\in A$ at time $t^{(i)}_{0}=0$ and jumps to state $x^{(i)}_{l}$ at time $t^{(i)}_{l}$, $1\leq l\leq\tau_{i}$, before it reaches set $B$ at time $t^{(i)}_{\tau_{i}}$ for the first time. That is, $x^{(i)}_{l}\in B^{c}$ for $0\leq l<\tau_{i}$ and $x^{(i)}_{\tau_{i}}\in B$. Also define $\sigma_{i}$ to be the last time when the $i$th trajectory visits set $A$. It satisfies $0\leq\sigma_{i}<\tau_{i}$, $x_{\sigma_{i}}^{(i)}\in A$ and $x_{l}^{(i)}\in A^{c}$ for $\sigma_{i}<l\leq\tau_{i}$. Then a natural way to estimate the function $\kappa$ and transition probabilities $p$ is given by

[TABLE]

where $x,y\in V$, provided that the denominator is nonzero. Similarly, for the initial distribution $\mu$, we can estimate

[TABLE]

where $x\in A$. Detailed studies related to the reconstruction of the Markov jump processes from data can be found in [22, 27] and references therein.

After estimating $\kappa$ and $p$ from (84), we can follow Algorithm 1 and the discussions in Subsection 5.1 to perform the analysis. In fact, direct calculation shows that function $\theta$ and flux $J$ in (35) and (40) are simply given by

[TABLE]

In other words, instead of solving the linear system (36), the ensemble average $\theta$ and $J$ of the constructed Markov jump process can be obtained by “counting” the trajectory data. However, we emphasize that this is not true in general for other quantities of ensemble averages. For instance, generally, the committor function $q$ and quantity $\bar{\theta}$, which satisfy equations (5) and (13) respectively, are different from

[TABLE]

which are computed by “counting” the trajectory data. As a simple counterexample, consider graph $G$ with four nodes $V=\{0,1,2,3\}$. We define sets $A=\{0\},B=\{3\}$ and suppose that only $2$ trajectories $(0,1,2,3)$, $(0,2,0,2,3)$ are observed (time information is ignored for simplicity). Then clearly $\sigma_{1}=0$, $\sigma_{2}=2$, and it follows from (87) that $q_{data}(1)=1$, $\bar{\theta}_{data}(1)=0$. On the other hand, from (84), we have $p(2\,|\,1)>0,p(0\,|\,2)>0$, and it follows that $q(1)<1$ as a consequence of Proposition 1. Similarly, we can also show that $\bar{\theta}(1)>0$.

In summary, after obtaining transition probabilities $p$ from (84), one still needs to apply Algorithm 1 to obtain various statistical quantities. In the first step of Algorithm 1, although function $\theta$ can be calculated from (86), it is necessary to compute the committor function $q$ by solving the linear system (5).

6 Numerical examples

In this section, we study several examples in order to demonstrate the analysis in the previous sections.

6.1 Example $1$ : continuous-time jump processes on a simple graph

As the first example, we consider continuous-time jump processes on a graph which consists of $7$ nodes. As shown in Figure 3 (a), we choose sets $A=\{1,2\}$, $B=\{7\}$, and study jump processes starting from set $A$ until they reach set $B$. We assume the initial distribution is uniform on $A$, i.e. $\mu(1)=\mu(2)=0.5$. Three different cases are considered, i.e. when the waiting time at each node satisfies (i) an exponential distribution, (ii) a power law distribution and (iii) a Weibull distribution, respectively (see Subsection 3.1).

In both cases of the exponential and Weibull distribution, we take the same rate constants $\lambda_{xy}$ in (47) and (53), which are shown in Figure 3 (a) for each edge $x\rightarrow y$, where $k=2$ is used for each edge in the Weibull distribution. In the case of the power law distribution, we choose $\alpha_{xy}$ in (55) to be equal to $\lambda_{xy}+1$ for each edge $x\rightarrow y$, such that the mean of the waiting time along each edge is the same as in the case of the exponential distribution.

In each case, starting from the continuous-time jump processes, we first construct the corresponding discrete-time jump processes using relations (42), (45) in Section 3. The transition probabilities $p$ for each edge as well as the committor function $q$ for each node are shown in Figure 3 (b)-(d). Clearly, different assumptions on the waiting time distributions of the continuous-time jump processes result in discrete-time Markov jump processes with different transition probabilities. In fact, from the formulas of transitions probabilities $p$ in (52), (54), we can conclude that, comparing to the exponential distribution case, the difference between jump probabilities $p$ along edge $x\rightarrow y$ and edge $x\rightarrow y^{\prime}$ where $\lambda_{xy}\neq\lambda_{xy^{\prime}}$ will be larger when the waiting times follow Weibull distribution with $k=2$. Furthermore, it can be observed that comparing to the exponential distributions, the probabilities to jump from “left to right” (e.g. along edges $3\rightarrow 5$, $4\rightarrow 6$ and $6\rightarrow 7$) are larger in the case of power law distributions, and are smaller in the case of Weibull distributions ($k=2$).

The ensembles of the nonreactive segments, the reactive segments, as well as the whole first passage paths are studied following the analysis in Section 2 and Section 3. The ensemble averages $\bar{\theta}^{\prime}$, $\widetilde{\theta}$, $\theta$, $\bar{T}$, $\widetilde{T}$ and $T$ for each node in different cases are shown in Table 2. Also see Figure 4 where these quantities are displayed in the case of the exponential distribution. We refer to equations (18), (27), (35), (57), (60), (63) as well as Table 1 for definitions of these quantities. In each case, from the values of the quantity $\bar{\theta}^{\prime}$ we can observe that the system will typically visit nodes $3,4$ and return to set $A$ a few times before it eventually arrives at set $B$. As shown in the columns with label “Total” in Table 2, we also see that while in each case the reactive trajectories contain a similar number of jumps on average ($4.60$–$5.95$, last column rows $2$, $9$, $16$), there are fewer jumps within the nonreactive trajectories in the case of power law distribution ($5.77$), but on the other hand there are more jumps in the nonreactive trajectories in the case of the Weibull distribution ($22.59$). The same is also true for the first passage paths as well as when we take time information into account. These results are accordant with our previous observations based on Figure 3 (b)-(d) that transitions of the system from set $A$ to $B$ are more difficult when the jump time follows the Weibull distribution.

6.2 Example $2$ : discrete-time random walk in a maze

In this subsection, we consider a (discrete-time) random walk in a maze. This example has been considered to illustrate TPT in [9]. However, since the theory there requires ergodicity of the process, an implicit assumption one need to make is that the walker keeps moving in the maze even after arriving at the exit cell. In this work, we investigate the same example (except we don’t have to make the aforementioned assumption) and apply the analysis in Section 2 to study its path ensembles.

The maze is constructed on a $30\times 30$ grid and the structure is shown in Figure 5. We assume that the walker enters the maze from the cell at the bottom left corner. At each unit of time, the walker moves to a new cell which is chosen randomly with equal probability from all neighboring cells that are directly reachable from the current one (no wall in between). The walker keeps moving until it finally arrives at the upper right corner where it can leave the maze.

To apply the analysis in Section 2, we construct an undirected graph whose nodes consist of the cells of the maze and two nodes are connected by an edge if and only if the corresponding cells in the maze are adjacent and there is no wall in between. Sets $A$, $B$ consist of a single node corresponding to the cell at the bottom left and the upper right corner (see Figure 5), respectively. Since the process itself is a discrete-time Markov jump process, its transition probabilities $p$ can be directly computed from the connectivity structure of the maze. Committor function $q$ and function $\theta$ related to the first passage paths are shown in Figure 6. From the committor function $q$, we can observe that there are long vertical walls which roughly separate the maze into left and right parts. In the left part (dark blue cells in the left panel of Figure 6), the walker is likely to return to $A$ first before it goes to $B$. On the other hand, from the plot of the function $\theta$ in the right panel of Figure 6, we can realize that, unlike the simple pathway shown in Figure 5, the first passage paths from $A$ to $B$ taken by the walker are very diffusive on average, since many cells have been repeatedly visited within a single first passage path. The diffusiveness of the paths can also be observed from the very long average path lengths in Table 3.

In order to further investigate the nonreactive and reactive segments of the first passage paths, functions $\bar{\theta}^{\prime}$, $\widetilde{\theta}$ are computed and are shown in Figure 7. We can see that both of these two segments are diffusive on average, since many cells are repeatedly visited. Comparing these two segments, however, it is observed that a large number of the visits at cells in the left part of the maze actually belong to the nonreactive segments and do not contribute to the reactive pathways. Comparing the two panels in Figure 7, we can also conclude that the nonreactive segments are typically more diffusive than the reactive segments. Roughly speaking, this phenomena occurs similarly when studying the transitions of a metastable diffusion process between two local minima of a potential, where the system spends longer time in the basin of attraction than in the transition region.

6.3 Example $3$ : discrete-time random walk on a scale-free network

In this example, we consider a network with a power law degree distribution. The network is generated using Python package NetworkX [11], which implemented the algorithm of Holme and Kim [14] to generate growing graphs with power law degree distribution. In order to have a better illustration, we choose a relatively small network with $50$ nodes. During the growth of the network, two random edges are added for each new node and the probability of adding a triangle after adding a random edge is $0.1$ [11]. The structure of the network is shown in Figure 8.

To continue, we take the discrete-time random walk on this network as an example and study its first passage paths starting from node $49$ to node [math], i.e. $A=\{49\},B=\{0\}$. Various statistics are computed following Algorithm 1, as well as the methods presented in Section 5. In Table 4, nodes are ranked according to different statistics and part of them are listed. It can be observed that, except for the source node $49$, the committor values $q$ of all other nodes are larger than $0.5$, reflecting the fact that these nodes are more tightly connected to node [math] compared to node $49$, which is the last node added during the growth of the network. We can also conclude that nodes with large values $\theta$ tend to have large values of $\bar{\theta}$ (or $\bar{\theta}^{\prime}$) and $\widetilde{\theta}$ as well. Furthermore, a positive correlation can be observed between the magnitudes of these statistics and the degrees of nodes, which probably could be explained by the undirected (symmetrical) movement of the random walker. In Figure 8, information related to the reactive trajectories are shown. Specifically, nodes with larger values $q$ are indicated by brighter colors, and nodes with larger values $\widetilde{\theta}$ are displayed in larger sizes. Also, edges with larger reactive fluxes $\widetilde{J}$ are plotted using thicker lines. From Figure 8, we could identify nodes and edges that are more frequently visited by reactive trajectories. The average path lengths of the nonreactive trajectories, the reactive trajectories, as well as the whole first passage paths are shown in Table 3. Different from the previous maze example, the lengths of the reactive trajectories are longer than those of the nonreactive trajectories on average, which is due to the fact that the attraction of the source set $A$ is not strong.

Summarizing the above results, we can conclude that, due to the global connectivity of the network, the path ensembles are widespread in path space and it is difficult to identify certain dominant transition pathways for this system. But still, we have obtained insights about the transitions of the random walk both quantitatively and graphically.

6.4 Example $4$ : football match

In the last example, we apply the analysis in Section 2 to study the football match between Germany and Argentina in the $2014$ world cup final [1]. The main aim is to quantify the performance of the players according to the ball passes during the match. For simplicity, we only focus on the German national team and model the ball passes between players as a discrete-time Markov jump process.

We take the data-based approach as discussed in Section 5. First, all passes between players of the Germany national team during the entire game ($90$ minutes plus $30$ minutes extra time) were recorded [2]. These passes constitute $107$ trajectories, each of which describes consecutive ball passes among German players when the team possessed the ball. Since each trajectory may start or finish in different ways, besides the $11$ starters and $2$ substitutes who participated in the game (see Table 5), $6$ additional nodes are introduced in the state set as outlined below. Specifically, a trajectory starts from node $1$ if the goalkeeper starts with a kick, and it starts from either node S[math] or node S$1$ when the Germany team got the possession from the opponent (or via a throw-in), depending on whether it occurred in the defensive Half or attacking Half. Similarly, for the end state, node L[math] and L$1$ indicate that the German team lost possession of the ball in the defensive Half and attacking Half, respectively. Furthermore, both nodes B[math] and B$1$ mark that the attack finished in the opponent’s penalty area, while the latter indicates a shot on goal was made.

We estimate the transition probabilities $p$ and the initial distribution $\mu$ of the graph using formulas (84), (85) in Section 5. Besides the three initial nodes S[math], S$1$ and $1$ (Neuer), we also select nodes $4$, $5$ and $20$ into the set $A$, since they correspond to the three full-back players (Höwedes, Hummels, Boateng) who stayed in the defensive Half most of the time during the match. Therefore, we have sets $A=\{\mbox{S}0\,,\mbox{S}1\,,1\,,4\,,5\,,20\}$ and the initial distribution for nodes $\{$S[math], S$1$, $1$, $4$, $5$, 20$$\} in set $A$ is $\mu=\{0.579,0.252,0.168,0,0,0\}$. Set $B$ is chosen to contain all end nodes of the trajectories, i.e. $B=\{\mbox{L}0\,,\mbox{L}1\,,\mbox{B}0\,,\mbox{B}1\}$. This choice of set $B$ allows us to utilize all the trajectories and the results can provide us an overall insights of this specific match, i.e. not only shots on goal but also attacks which were not successful.

After these preparations, we apply Algorithm 1 to compute the various statistics defined in Section 2 and results are shown in Figure 9 and Figure 10. As already mentioned in Section 5, function $\theta$, flux $J$ and the average total length of the first passage paths computed from Algorithm 1 are identical to those computed from estimators in (86). In Figure 9, nodes and edges are displayed in different sizes and thickness according to the values of function $\theta$ and flux $J$. It could be observed that the four players Schweinsteiger, Kroos, Lahm, Özil had much possession of the ball and contributed more ball passes compared to the other players. In Figure 10, on the other hand, the sizes of nodes and the thickness of edges are determined according to their values of function $\widetilde{\theta}$ and $\widetilde{J}$, respectively, which are related to the reactive trajectory ensemble. We see that passes among the goalkeeper and the full-back players (Neuer, Höwedes, Hummels, Boateng) are filtered out. Furthermore, upon closer examination of the edges, we can observe that the role of the attackers (e.g. Özil, Müller) become more prominent. Quantitative results are presented in Table 6.

7 Conclusions

In the present work, we have developed a theory for the statistics of the first passage path ensemble of jump processes on a finite state space. The main approach is to divide a first passage path into nonreactive and reactive segments so that the behaviors of the processes within each segment can be studied separately. Furthermore, relations between the statistics of these two segments and the statistics of the entire first passage paths are obtained.

Our analysis can be applied to jump processes which are non-ergodic, as well as continuous-time jump processes where the waiting time distributions are non-exponential. More generally, second-order or higher order Markov chains have been studied where the dynamics depend not only on system’s current state but also on the states which have been visited in the previous steps [28, 29, 31]. These types of jump processes can be converted to Markov jump processes by extending the state space and therefore it is possible to apply the analysis in the current work to study these higher order Markov chains (although the size of the extended state space may become quite large and brings computational difficulties).

We expect that the study of both the nonreactive and reactive segments can help to understand the transition behavior of processes from one subset to another. Especially, analysis of the reactive segments in this work is closely related to TPT, which was developed for both diffusion processes and jump processes under the assumption of ergodicity. Some illustrative examples are studied numerically in order to demonstrate the applicability of the theory. More generally, the study of the transition events, or the first passage phenomena [21], plays an important role in order to understand many real-world systems and applications, e.g. in molecular dynamics or in epidemiology. Applying the analysis of the current paper to study phenomena in the aforementioned subject areas will be considered in future work.

Acknowledgement

This research has been funded by Deutsche Forschungsgemeinschaft (DFG) through grant CRC 1114, by the Einstein Foundation Berlin through project CH4 of Einstein Center for Mathematics (ECMath) and though the BMBF, grant number 031A307.

Appendix A An alternative expression of $\mu_{r}$

Here we provide an alternative way of studying the probability distribution $\mu_{r}$ defined in (17) in Section 2. Consider the first passage paths starting from $x\in B^{c}$, and for each $y\in A$, let $\omega(x,y)$ be the probability that $y$ is the last hitting node in $A$, i.e. $\omega(x,y)=\mathbb{P}(x_{\sigma_{x}}=y)$. The equation of $\omega$ can be obtained by considering the next state $z$ which the system will jump to from $x$. In the case $x\neq y$, we obtain the relation

[TABLE]

Notice that for $x\not\in V^{-}$, we have $\omega(x,y)=0$ and (88) still holds.

In the case $x=y\in A$, after jumping to another state $z$, there are two possibilities depending on whether the system will return to set $A$ again before it reaches set $B$. Notice that such events are characterized exactly by $\{\tau_{A,z}<\tau_{B,z}\}$ and its complement. Using the committor function $q$ in (4), (5), we obtain

[TABLE]

For each $y\in A$, we can solve $\omega(\cdot,y)$ from (88)-(89). Since the initial distribution is $\mu$ in the first passage path ensemble, we obtain

[TABLE]

Appendix B Path probabilities for a continuous-time jump process

In the following, starting from a continuous-time jump process, we study the probability associated to its jump trajectories and further clarify the connection between the continuous-time and discrete-time (Markov) jump process defined by transition probabilities $p$ in Subsection 3.1.

Following the setup in Subsection 3.1, let $\rho(t,x)$ and $w(t,x)$ denote the probabilities that the system stays at node $x$ at time $t$ and the system jumps to node $x$ (from another node) at time $t$, respectively. They are related by

[TABLE]

and we have $w(0,x)=\rho(0,x)$ at time $t=0$. Considering the node $y$ from which the system jumps to $x$, we can obtain the equation

[TABLE]

Expressions of $\rho(t,x),w(t,x)$ have been obtained in [13] by applying Laplace transformation in (91) and (92).

A trajectory of the continuous-time jump process can be represented as a path with time-stamps

[TABLE]

where $t_{0}=0$ and $0\leq t_{1}<\cdots<t_{N}$ are the jump time, $x_{k}\in V$, $0\leq k\leq N$. It corresponds to the event that the process starts from state $x_{0}$ at time $t_{0}=0$ and later on jumps consecutively from node $x_{k}$ to $x_{k+1}$ at time $t_{k+1}$, $0\leq k\leq N-1$. The probability density that this specific path $\varphi$ occurs is

[TABLE]

Now we consider the path $\varphi$ when the sequence of nodes $(x_{0},x_{1},\cdots,x_{N})$ is fixed and the jump times are allowed to vary. For $1\leq k\leq N$, we denote the probability density that the system arrives at node $x_{k}$ at time $t$ along the sequence of nodes $(x_{0},x_{1},\cdots,x_{k})$ by $r_{k}(t)$. Also, let $\eta(t)$ be the probability that the system arrives at state $x_{N}$ along the path $(x_{0},x_{1},\cdots,x_{N})$ and remains there until time $t$. Notice that we have omitted the dependence on this specific node sequence in the notations of $r_{k}(\cdot)$ and $\eta(\cdot)$ for simplicity. Setting $b_{k}(\cdot)=b(\cdot\,,x_{k}\rightarrow x_{k+1})$ for $0\leq k<N$, then clearly we have $r_{1}(t)=b_{0}(t)$ and

[TABLE]

for $2\leq k\leq N$, and also $\eta(t)=\int_{0}^{t}r_{N}(u)\,a(t-u\,|\,x_{N})\,du$. From (95), it is direct to verify that

[TABLE]

and therefore we can obtain

[TABLE]

where $\hat{r}_{k}$, $\hat{\eta}$, $\hat{b}_{k}$, $\hat{a}$ denote the Laplace transformations of functions $r_{k},\eta$, $b_{k}$ and $a(\cdot\,|\,x_{N})$, respectively. As a result, we obtain the expressions

[TABLE]

where $1\leq k\leq N$. When the probability densities $\psi$ of the waiting times are exponential distributions, (98) can be further simplified and the expression of $\eta(t)$ has been obtained in [33]. Also see [12] for a sampling algorithm based on these quantities.

From (95) and (45), we can also compute the probability that the system arrives at $x_{N}$ along the specific path $x_{0},x_{1},\cdots,x_{N}$ (regardless of the jump times) and obtain

[TABLE]

The right hand side of the expression above indicates that we can focus on the discrete-time Markov jump process on $G$ defined by the transition probability matrix $P$, whose entries are $P_{xy}=p(y\,|\,x)$, if we are interested in the jump paths irrespective of the time information.