How fast can we reach a target vertex in stochastic temporal graphs?

TL;DR

This paper investigates the complexity of reaching a target vertex in stochastic temporal graphs with different memory effects, revealing that certain path problems are computationally hard or efficiently solvable depending on the memory model.

Contribution

It introduces a hierarchy of memory-$k$ models in stochastic temporal graphs and analyzes the computational complexity of two key path problems across these models.

Findings

MINIMUM ARRIVAL is #P-hard for memory-0 models.

BEST POLICY can be solved in polynomial time for memory-0 models.

Memory effects significantly impact the computational complexity of path problems.

Abstract

Temporal graphs are used to abstractly model real-life networks that are inherently dynamic in nature. Given a static underlying graph $G=(V,E)$, a temporal graph on $G$ is a sequence of snapshots $G_t$, one for each time step $t\geq 1$. In this paper we study stochastic temporal graphs, i.e. stochastic processes $\mathcal{G}$ whose random variables are the snapshots of a temporal graph on $G$. A natural feature observed in various real-life scenarios is a memory effect in the appearance probabilities of particular edges; i.e. the probability an edge $e\in E$ appears at time step $t$ depends on its appearance (or absence) at the previous $k$ steps. In this paper we study the hierarchy of models memory-$k$, addressing this memory effect in an edge-centric network evolution: every edge of $G$ has its own independent probability distribution for its appearance over time. Clearly, for every $k\geq 1$, memory-$(k-1)$ is a special case of memory-$k$. We make a clear distinction between the values $k=0$ ("no memory") and $k\geq 1$ ("some memory"), as in some cases these models exhibit a fundamentally different computational behavior, as our results indicate. For every $k\geq 0$ we investigate the complexity of two naturally related, but fundamentally different, temporal path (journey) problems: MINIMUM ARRIVAL and BEST POLICY. In the first problem we are looking for the expected arrival time of a foremost journey between two designated vertices $s,y$. In the second one we are looking for the arrival time of the best policy for actually choosing a particular $s$-$y$ journey. We present a detailed investigation of the computational landscape of both problems for the different values of memory $k$. Among other results we prove that, surprisingly, MINIMUM ARRIVAL is strictly harder than BEST POLICY; in fact, for $k=0$, MINIMUM ARRIVAL is #P-hard while BEST POLICY is solvable in $O(n^2)$ time.

Equations

$$M_{e}=\left(\begin{array}[]{r|cc}&0&1\\ \hline\cr 0&1-p_{e}&p_{e}\\ 1&q_{e}&1-q_{e}\end{array}\right)\text{, where }0\leq p_{e},q_{e}\leq 1.$$

$$X(a,c)>k\iff T_b>k\text{ and }{T}_d>k$$

$$\mathrm{Pr}[T_b>k]=\frac{1}{2^k}+k\frac{1}{2}\frac{1}{2^{k-1}}=(k+1)\frac{1}{2^k}.$$

$$\mathrm{Pr}[X(a,c)>k]=\mathrm{Pr}[T_b>k]\mathrm{Pr}[T_d>k]=\frac{1}{2^{2k}}+k\frac{1}{2^{2k-1}}+k^2\frac{1}{2^{2k}}.$$

$$\Phi =\underset{(i,j)\in E}{\bigvee }{x}_i{y}_j,$$

$$\mathbb{E}[X(s,y)]=3p+4(1-p)=4-p,$$

$$\mu =\sum _{i=1}^{\infty }\mathrm{Pr}[X_{{\mathcal{P}}^{(0)}}(s,y)\ge i].$$

$$\mathrm{Pr}[X\ge \lambda \mu ]\le e^{1-\lambda }.$$

$$\mu -\sum _{i=1}^{\tau }\mathrm{Pr}[X_{{\mathcal{P}}^{(0)}}(s,y)\ge i]=\sum _{i=\tau +1}^{\infty }\mathrm{Pr}[X_{{\mathcal{P}}^{(0)}}(s,y)\ge i]<\epsilon ,$$

$$\begin{array}{rl}\sum _{i=\tau +1}^{\infty }\mathrm{Pr}[X_{{\mathcal{P}}^{(0)}}(s,y)\ge i]& \le \sum _{i=\tau +1}^{\infty }{e}^{1-\frac{i}{\mu }}=\frac{e^{1-\tau /\mu }}{e^{1/\mu }-1}\le \frac{e^{1-\mu \left(\ln \frac{\mu }{\epsilon }+1\right)/\mu }}{e^{1/\mu }-1}=\\ & =\frac{\epsilon }{\mu (e^{1/\mu }-1)}\le \frac{\epsilon }{\mu (1+\frac{1}{\mu }+\frac{1}{2{\mu }^2}-1)}=\frac{\epsilon }{1+\frac{1}{2\mu }}<\epsilon ,\end{array}$$

$$O\Big{(}f\left(n^{c+1}\ln\frac{n}{\varepsilon},n,m\right)+n\ln n+m\Big{)}.$$

$$\sum _{i=\tau +1}^{\infty }\mathrm{Pr}[X_{{\mathcal{G}}^{(0)}}(s,y)\ge i]\le \sum _{i=\tau +1}^{\infty }\mathrm{Pr}[X_{{\mathcal{P}}^{(0)}}(s,y)\ge i]<\epsilon ,$$

$$\displaystyle\sum_{i=1}^{\tau}\Pr[X_{\mathcal{G}^{(0)}}({s},{y})\geq i]\$$

$$\mathrm{Pr}[X_{{\mathcal{G}}^{(0)}}(s,y)\ge i],i=1,\dots ,\ell .$$

$$\mathrm{Pr}[X_{{\mathcal{G}}^{(0)}}(s,y)=i],i=1,\dots ,\ell -1.$$

$$\mathrm{Pr}[X_{{\mathcal{G}}^{(0)}}(s,y)=i]=(1-p{)}^{i-1}p,$$

$$\mathrm{Pr}[X_{{\mathcal{G}}^{(0)}}(s,y)\ge i]=\mathrm{Pr}[X_{{\mathcal{H}}_1^{(0)}}(s_1,y_1)\ge i]\cdot \mathrm{Pr}[X_{{\mathcal{H}}_2^{(0)}}(s_2,y_2)\ge i],$$

$$\mathrm{Pr}[X_{{\mathcal{G}}^{(0)}}(s,y)=i]=\sum _{j=1}^{i-1}\mathrm{Pr}[X_{{\mathcal{H}}_1^{(0)}}(s_1,y_1)=j]\cdot \mathrm{Pr}[X_{{\mathcal{H}}_2^{(0)}}(s_2,y_2)=i-j].$$

$$\mathrm{Pr}[X_e^t\ge a]$$

$$\mathbb{E}[X(s,y)]=\sum _{i=1}^{\infty }\mathrm{Pr}[X(s,y)\ge i]\le \sum _{i=1}^{\infty }Pr\left[nZ\ge a\right]=\mathbb{E}[(n-1)Z]\le n^{c+1}.$$

$$\mathrm{Pr}\left[|\tilde{X}-\mathbb{E}[X]|\ge \epsilon \mathbb{E}[X]\right]\le {\left(\frac{\sigma (\tilde{X})}{\epsilon \mathbb{E}[X]}\right)}^2.$$

$$\mathrm{Pr}\left[|\tilde{X}-\mathbb{E}[X]|\ge \epsilon \mathbb{E}[X]\right]\le \frac{{\sigma }^2(X)}{{\epsilon }^{2}r{\mathbb{E}}^2[X]}\le \frac{{\sigma }^2(X)}{{\epsilon }^{2}r}.$$

$${\sigma }^2(X)=\mathbb{E}[X^2]-{\mathbb{E}}^2[X]\le \mathbb{E}[X^2]\text{leavevmodenobreak (since }\mathbb{E}[X]\ge 0\text{).}$$

$$X^2\le (Y_1+\dots +Y_{\ell }{)}^2\le (1^2+\dots +1^2)\cdot (Y_1^2+\dots +Y_{\ell }^2)\le n(Y_1^2+\dots +Y_{\ell }^2).$$

$${\sigma }^2(X)\le 2n^{2c+2}.$$

$$\mathrm{Pr}\left[|\tilde{X}-\mathbb{E}[X]|\ge \epsilon \mathbb{E}[X]\right]\le 2\frac{n^{2c+2}}{{\epsilon }^{2}r}.$$

$$\mathrm{Pr}\left[\tilde{X}\in (1-\epsilon ,1+\epsilon )\cdot \mathbb{E}[X]\right]\ge 1-\frac{1}{n},$$

$$\mathrm{Pr}[\text{failure of FPRAS}]$$

$$h(v_i)=\frac{{\sum }_{j=1}^{i-1}{Q}_{i,j}h(v_j)+1}{1-Q_i},$$

$$M(1-g)$$

Full text

Keywords: Temporal network, stochastic temporal graph, temporal path, #P-hard problem, polynomial-time approximation scheme.

1 Introduction

Dynamic network analysis, i.e. analysis of networks that change over time, is currently one of the most active topics of research in network science and theory. A common task in this field is to use our prior knowledge of the network link dynamics to answer questions about the behavior of the network over time, e.g. how quickly information can flow through it. Many modern real-life networks are dynamic in nature, in the sense that the network structure undergoes discrete changes over time [30, 36]. Here we deal with the discrete-time dynamicity of the network links (edges) over a fixed set of nodes (vertices). That is, given an underlying static graph $G$, the network evolution over $G$ is given by the successive appearance or absence of each edge of $G$ at every time step $t=1,2,\ldots$. This concept of dynamic network evolution is given by temporal graphs [26, 28], which are also known by other names such as evolving graphs [5, 19], or time-varying graphs [1]. For a recent attempt to integrate existing models, concepts, and results from the distributed computing perspective, see the survey papers [11, 12] and the references therein.

Definition 1** (Temporal graph).**

Given an underlying static graph $G=(V,E)$ on $n$ vertices and $m$ edges, a temporal graph on $G$ is a sequence $\mathcal{G}=\{G_{t}=(V,E_{t}):t\in\mathbb{N}\}$ of graphs such that $E_{t}\subseteq E$ for all $t\in\mathbb{N}$. Every $G_{t}$ is the snapshot of $\mathcal{G}$ at time step $t$.

Another way to think about temporal graphs is by assigning time-labels on the edges; for example, if an edge $e$ appears in the snapshots $G_{3}$, $G_{5}$, and $G_{8}$, then we equivalently assign to $e$ the set of labels $\lambda(e)=\{3,5,8\}$. Due to the vast applicability of temporal graphs, various structural and algorithmic properties of them have been studied extensively, both via theoretical/algorithmic analysis and via empirical simulation-based analysis. In many of these works, one of the central temporal notions is that of a temporal path. A path in the underlying (static) graph $G$ is a temporal path (or journey) if there exists an increasing sequence of time-labels as one walks along the edges of the path [26, 28]. Motivated by the fact that, due to causality, information in temporal graphs can only flow along sequences of edges that appear in an increasing time order, many temporal graph parameters and optimization problems that have been studied so far are based on the notion of a temporal path and other related notions, e.g. temporal analogs of distance, diameter, connectivity, reachability, and exploration [3, 2, 22, 33, 9, 7, 13, 18, 20, 6, 27, 17]. In addition to temporal paths, recently also various temporal non-path problems have been introduced and algorithmically studied, such as temporal vertex cover [4], temporal coloring [29], and temporal $\Delta$-cliques [41, 23].

Apart from the focus on the various algorithmic problems that one can study on temporal graphs, one can also view temporal graphs through several different levels of knowledge about the actual network evolution. On the one extreme, we may be given the whole temporal graph instance in advance, i.e. the times of appearance and absence of every edge at all times, as it typically happens e.g. when modeling transportation networks. On the other extreme, the temporal graph may be created by an adversary who reveals it to us snapshot-by-snapshot at every time step. Here we focus on the intermediate knowledge settings, captured by stochastic temporal graphs, where the network evolution is given by a probability distribution that governs the appearance of each edge over time.

Definition 2** (Stochastic temporal graph).**

A stochastic temporal graph is a stochastic process $\mathcal{G}=\{G_{t}:t\in\mathbb{N}\}$ whose random variables are snapshots $G_{t}\subseteq G$ of an underlying graph $G$. Every instantiation of $\mathcal{G}$ is a temporal graph.

A natural feature of stochastic temporal graphs which can be observed in various real-life scenarios (and which we address in this paper) is that the appearance probability of a particular edge at a given time step $t$ depends on the appearance (or absence) of the same edge at the previous $k\geq 1$ time steps. This “memory effect” can often be observed, among others, in faulty network communication and in mobile, social, and peer-to-peer networks [14, 37, 34]. Several other models of temporal networks which exhibit some sort of probabilistic behavior have been considered in the past, see e.g. [24].

In this paper, we study a hierarchy of models for stochastic temporal graphs. These models concern an edge-centric network evolution, i.e. they assign to every edge of the underlying graph $G$ a probability distribution for its appearance over time, independently of all the other edges. The first and most basic model (memoryless or memory-[math]) assigns independently to every edge $e$ a probability $p_{e}$ such that, at every time step, $e$ appears with probability $p_{e}$. In the general model (memory-$k$), at every time step the appearance probability of every edge is a function of the history of its appearances/absences in the last $k\geq 1$ time steps. Clearly, for every $k\geq 1$, the memory-$(k-1)$ model is a special case of the memory-$k$ model. However, in this paper we make a clear distinction between the values $k=0$ (“no memory”) and $k\geq 1$ (“some memory”), as in some cases these models exhibit a fundamentally different computational behavior for these values of $k$, as our results indicate (see Section 4).

Our memory-$k$ model, $k\geq 1$, is a direct generalization of the homogeneous version of the memory-1 model that was introduced in a seminal paper by Clementi et al. [15], in which all edges have the same probability distribution for their appearance, based on their own appearance/absence at the previous step. In this homogeneous memory-1 model, Clementi et al. gave upper bounds for the flooding time and they provided tight characterizations of the graphs on which the flooding time is constant [15]. It is worth noting here that Avin et al. [6] studied the completely opposite extreme of our edge-centric evolution; namely they considered a graph-centric evolution model where a global probability distribution assigns specific transition probabilities among different snapshots [6]. Between the two extremes of the edge-centric and the graph-centric network evolution models, there exists a whole hierarchy of locally interdependent probabilistic patterns, i.e. probability distributions where the appearance probability of one edge also depends on the appearance of other edges over time; such models remain mostly unexplored.

In both our memoryless and memory-$k$ variations of stochastic temporal graphs, we study two fundamental temporal path (i.e. journey) problems that are defined on two designated vertices ${s}$ and ${y}$. Consider a piece of information that is generated at ${s}$ at time 1, which we would like to send to ${y}$ via an ${s}$-${y}$ journey. The arrival time of an ${s}$-${y}$ journey in a realization of a stochastic temporal graph is the time the information reaches ${y}$ using this journey. A foremost ${s}$-${y}$ journey is one with the smallest arrival time. In the first part of the paper we investigate the complexity of computing the expected arrival time of a foremost ${s}$-${y}$ journey. Basu et al. [8] and Nain et al. [31] studied a similar problem but their work is restricted to the simpler cases where the underlying graph is either a path or a grid.

In the second part of the paper we investigate the complexity of computing the arrival time of a best policy for actually choosing a particular ${s}$-${y}$ journey in the stochastic temporal graph. To illustrate this notion of a best policy, assume that some piece of information is carried by an entity, say Alice. Alice is given as input the parameters of the stochastic temporal graph (i.e. the probabilistic rules on the edges) and, at every time step, she knows the current snapshot and her current location. Based on this information, Alice has to decide at every step for her next action, while her goal is to reach ${y}$ as quickly as possible on expectation, starting at time 1. In a very inspiring paper, Basu et al. [7] consider this problem in the special case of the memoryless model where all edges have the same probability of appearance at every time, and give a Dijkstra-like polynomial-time algorithm. Special cases of the memory-1 model were considered in [10].

To illustrate the difference between the two problems we study, we make the following analogy. In the first problem (Minimum Arrival) we try to transfer information from ${s}$ to ${y}$ using an unbounded number of messages, i.e. we “flood” the stochastic temporal graph with information. Initially the information is stored at ${s}$ at time 1 and then, at every step, every informed vertex informs all its neighbors as soon as the edge between them becomes available. In the second problem (Best Policy) we try to transfer a package with a tangible good from ${s}$ to ${y}$. Now, at every step we need to decide for the actual route of the package through the network: when an edge appears, should we ship the package along it or rather wait where we currently are? Best Policy is more relevant to real-life applications than Minimum Arrival, where an actual good journey needs to be found in real time.

Our contribution. In the first part of the paper, in Section 3, we provide our results for the problem Minimum Arrival, i.e. for computing the expected arrival time of a foremost ${s}$-${y}$ journey in a stochastic temporal graph. First we prove in Section 3.1 that Minimum Arrival is #P-hard even for the memoryless model (and thus also for the memory-$k$ model, for every $k\geq 1$). The reduction is done from the problem #PP2DNF which counts the number of satisfying assignments in a positive partitioned 2-DNF Boolean formula [35].

Second, we provide in Section 3.2 a non-trivial approximation scheme for Minimum Arrival, based on dynamic programming, for the memoryless model in the case where the underlying graph $G$ is a series-parallel graph. More specifically, it turns out that this is a Fully Polynomial-Time Approximation Scheme (FPTAS) whenever the probabilities $p_{e}$ are lower bounded by $\frac{1}{n^{c}}$ for some $c\geq 1$. Let $X$ be the random variable that expresses the arrival time of a foremost ${s}$-${y}$ journey. For every $\varepsilon\in(0,1]$, our FPTAS gives an algorithm that produces a value $\mu$ where $\mathbb{E}(X)-\varepsilon\leq\mu\leq\mathbb{E}(X)$, and runs in polynomial time in both $n$ and $\frac{1}{\varepsilon}$. Although our main result of Section 3.2 concerns series-parallel graphs, we actually present a more general FPTAS approach (see Theorem 3) which is of independent interest and could lead to FPTASs also for more general classes of underlying graphs $G$.

Third, we present in Section 3.3 a Fully Polynomial Randomized Approximation Scheme (FPRAS) for Minimum Arrival in the memory-$k$ model, for every $k\geq 0$, under the assumption that every edge appearance probability is lower bounded by $\frac{1}{n^{c}}$ for some $c\geq 1$. Let $X$ be the random variable that expresses the arrival time of a foremost ${s}$-${y}$ journey. For every $\varepsilon\in(0,1)$, our FPRAS gives a randomized algorithm that produces an estimate $\widetilde{X}$ where $(1-\varepsilon)\mathbb{E}(X)\leq\widetilde{X}\leq(1+\varepsilon)\mathbb{E}(X)$ with probability tending to 1 as $n\rightarrow\infty$, and runs in polynomial time in both $n$ and $\frac{1}{\varepsilon}$.

In the second part of the paper, in Section 4, we provide our results for the problem Best Policy, i.e. for computing the expected arrival time of a best policy for choosing a particular ${s}$-${y}$ journey. Initially we provide in Section 4.1 a dynamic programming algorithm for the memoryless model which runs in $O(n^{2})$ time and space. In wide contrast, we prove in Section 4.2 that Best Policy becomes #P-hard for the memory-$k$ model, where $k\geq 3$, again by providing a reduction from the problem #PP2DNF. Finally, we provide in Section 4.3 a formulation of Best Policy in the memory-$k$ model using the general Markov Decision Process (MDP) framework which allows us to devise in Section 4.3.2 an exact doubly exponential-time algorithm with running time $O(2^{(kmn+n\log n)\cdot 2^{km}})$.

2 Preliminaries

In this paper we consider temporal graphs (see Definition 1) in which the underlying (static) graph $G=(V,E)$ has $n$ vertices and $m$ edges. A subgraph $H=(V,E_{H})$ of $G$, denoted by $H\subseteq G$, is a graph where $E_{H}\subseteq E$. For every vertex $u\in V$, the neighborhood $\Gamma_{G}(u)$ of $u$ in $G$ is the set of adjacent vertices of $u$ in $G$. The closed neighborhood $\Gamma_{G}[u]$ also contains vertex $u$ itself, i.e. $\Gamma_{G}[u]=\Gamma_{G}(u)\cup\{u\}$. For simplicity of notation we denote $[n]=\{1,2,\ldots,n\}$ for every $n\in\mathbb{N}$. Furthermore, sometimes we refer to the discrete time steps $t=1,2,\ldots$ as days. Throughout the paper we consider stochastic temporal graphs that exhibit an edge-centric evolution, i.e. every edge $e$ of $G$ is assigned one probability distribution for its appearance over time, independently of all other edges. We investigate the case where there is a “memory effect” that governs the probability of appearance of every edge over time. We distinguish now the cases where the the memory is zero or non-zero.

Memoryless (or memory-0) model.

Every edge $e\in E$ evolves stochastically and independently of other edges as follows: at every time step $t\in\mathbb{N}$, $e$ appears in $G_{t}$ with probability $p_{e}$ and is absent with probability $1-p_{e}$, independently of any other time step. The numbers $\{p_{e}:e\in E\}$ are given parameters of the model. We denote this (memoryless) stochastic temporal graph by $\mathcal{G}^{(0)}=(G,\{p_{e}:e\in E\})$ or simply $\mathcal{G}^{(0)}=(G,\{p_{e}\})$.

Memory-$k$ model.

This model of temporal graphs exhibits stochastic time-dependency of the edges: we assume an initial (arbitrary) sequence of $k$ snapshots, $G_{-k+1},\ldots,G_{-1},G_{0}\subseteq G$. At every time step $t\geq 1$, every edge $e$ appears independently of all other edges with probability that depends only on (the edge and) the history of appearance of $e$ in the $k$ previous snapshots. At every time step $t$, this history is a $k$-bit binary vector, where a [math]-entry (resp. $1$-entry) on the $i$-th position denotes absence (resp. appearance) of $e$ in $E_{t-k+i-1}$, for $i=1,\ldots,k$. Therefore the snapshot $G_{t}$ is the graph that appears at time $t\geq 1$ as the result of the following experiment: given the history $H_{e}^{(k)}$ of the appearance of edge $e\in E$ in the last $k$ snapshots, $e$ belongs to $E_{t}$ independently with probability $p_{e}(H_{e}^{(k)})$. We denote the memory-$k$ stochastic temporal graph by $\mathcal{G}^{(k)}$.

In the particular case where $k=1$, the memory-$1$ stochastic temporal graph $\mathcal{G}^{(1)}$ is the sequence $\{G_{t}=(V,E_{t}):t\in\mathbb{N}\}$ of snapshots such that $E_{t}=\{e\in E:X_{t}^{e}=1\}$, where $\{X_{t}^{e}\}_{t\in\mathbb{N}}$ is a Markov chain for the edge $e\in E$ with states $\{0,1\}$ (corresponding to non-appearance and appearance of $e$, respectively) and probability transition matrix:

[TABLE]

Using this formalism, $p_{e}$ (resp. $q_{e}$) is the probability that the edge $e$ changes its current state from absence to appearance (resp. from appearance to absence) in the next snapshot. Note here that, setting $p_{e}=p$ and $q_{e}=q$ for every edge $e$, we obtain exactly the well-established edge-Markovian evolving graph model introduced by Clementi et al. [15].

2.1 The problems

This work studies two main problems, each under the models of stochastic temporal graphs defined above. To describe both of these problems, let us first recall that information in temporal graphs flows via journeys, i.e. temporal paths.

Definition 3** (Time-edge).**

A time-edge in a temporal graph $\mathcal{G}=\{G_{t}:t\in\mathbb{N}\}$ is a pair $(e,t)$ such that $e\in E_{t}$.

Definition 4** (Journey / temporal path).**

Let $\mathcal{G}=\{G_{t}:t\in\mathbb{N}\}$ be a temporal graph and ${s},{y}$ be two vertices of $G$. An ${s}$-${y}$ journey (or an ${s}$-${y}$ temporal path) in $\mathcal{G}$ is a sequence \big{(}(e_{1},t_{1}),\ldots,(e_{x},t_{x})\big{)} of time-edges over a path $(e_{1},\ldots,e_{x})$ in $G$, where $t_{1}<t_{2}<\ldots<t_{x}$. The arrival time of the journey is the time $t_{x}$ of appearance of its last edge.

Definition 5** (Foremost Journey).**

A foremost ${s}$-${y}$ journey in a temporal graph $\mathcal{G}$ is an ${s}$-${y}$ journey with minimum arrival time amongst all ${s}$-${y}$ journeys in $\mathcal{G}$.

Notice that the arrival time of a foremost ${s}$-${y}$ journey in a stochastic temporal graph is a random variable, which we henceforth denote by $X({s},{y})$. The first problem that we study here is how to compute the expected value of the latter, namely $\mathbb{E}[X({s},{y})]$.

Problem 1** (Minimum Arrival).**

Given a stochastic temporal graph on an underlying graph $G=(V,E)$ and two distinct vertices ${s},{y}\in V$, compute the expected value of the arrival time of a foremost ${s}$-${y}$ journey, i.e. $\mathbb{E}[X({s},{y})]$.

Now suppose that an individual (say Alice) is at day 0 at vertex ${s}$ and would like to arrive at vertex ${y}$ through a temporal path as quickly as possible. Denote by $s_{t}$ the vertex where she is located at time $t$; then $s_{0}={s}$. Every day $t$ Alice “wakes up” in the morning and looks at which edges are available in today’s snapshot; by only knowing her current position, the history of the last $k$ snapshots, and the input parameters of the stochastic temporal graph (i.e. the probabilistic rules of edge appearance), Alice needs to decide whether:

1. (i)

to stay at the vertex $s_{t}$ she currently is, or 2. (ii)

to use an edge of $G_{t}$ to move to a neighboring vertex.

That is, $s_{t+1}$ is either equal to $s_{t}$ or equal to some vertex of $\Gamma_{G_{t}}(s_{t})$.

A natural problem we can study here is to compute the expected arrival time of an ${s}$-${y}$ journey that Alice can follow, using a best policy111We use the term “policy” here (instead of “strategy”) since, as we will see later, this problem can be formulated using a Markov Decision Process (MDP). possible, i.e. a policy (sequence of actions) that minimizes her expected arrival time at ${y}$. Notice that the arrival time of the journey suggested to Alice by the best policy is a random variable $Y({s},{y})$, whose distribution depends on the specific stochastic temporal graph. In particular, in the memoryless model, the expectation of $Y({s},{y})$ depends only on the edges’ probabilities of appearance. In the memory-$k$ model, the expectation of $Y({s},{y})$ also depends on the initial snapshots $G_{-k+1},\ldots,G_{-1},G_{0}$.

Problem 2** (Best Policy).**

Given a stochastic temporal graph $\mathcal{G}^{(k)}$ on an underlying graph $G=(V,E)$ and two distinct vertices ${s},{y}\in V$, compute $\mathbb{E}_{\mathcal{G}^{(k)}}[Y({s},{y})]$.

In particular, we will write $h({s},{y})\stackrel{{\scriptstyle\text{def}}}{{=}}\mathbb{E}_{\mathcal{G}^{(0)}}[Y({s},{y})]$ and $h({s},{y},G_{0})\stackrel{{\scriptstyle\text{def}}}{{=}}\mathbb{E}_{\mathcal{G}^{(1)}}[Y({s},{y})]$.

Difference between the two problems.

Before we proceed further, we first give an example illustrating that the problems Minimum Arrival and Best Policy are different. To demonstrate this, assume the memoryless model $\mathcal{G}^{(0)}$ and consider the 4-cycle $a,b,c,d,a$ as the underlying graph. Let ${s}=a$ and ${y}=c$ and assume that, at any time step, each edge appears independently with probability $\frac{1}{2}$.

Any best policy for Alice will wait until an edge incident to $a$ appears and then cross it; if both adjacent edges $(a,b)$ and $(a,d)$ appear at the same time, then it does not matter which one she chooses. The event “some edge adjacent to $a$ appears” occurs with probability $\frac{3}{4}$, hence, the expected time until such an edge appears is $\frac{4}{3}$. Furthermore, when Alice reaches one of the vertices $b$ or $d$, an optimal policy will never suggest going back to $a$, so Alice will have to wait until the last edge to $c$ appears, which takes $2$ steps on expectation. Overall, the optimal policy for Alice will take $h(a,c)=\frac{10}{3}$ steps on expectation. This is the solution to Best Policy (see Problem 2).

On the other hand, Minimum Arrival (see Problem 1) asks for the expectation of the arrival time $X(a,c)$ of a foremost ${s}$-${y}$ journey. To compute $\mathbb{E}[X(a,c)]$, denote by $T_{b}$ (resp. $T_{d}$) the arrival time of a journey allowed to use only edges $(a,b)$ and $(b,c)$ (resp. $(a,d)$ and $(d,c)$), when they appear. Then,

[TABLE]

But the probability of the event $\{T_{b}>k\}$ is equal to the probability that either $(a,b)$ does not appear until (and including) step $k$ plus the probability that it appears within the first $k$ steps, and $(b,c)$ does not appear after that until (and including) $k$. Therefore,

[TABLE]

By symmetry we have $\Pr[T_{b}>k]=\Pr[T_{d}>k]$ and, by independence, for any $k\geq 2$:

[TABLE]

By using the fact that $\mathbb{E}[X(a,c)]=\sum_{k=0}^{\infty}\Pr[X(a,c)>k]=2+\sum_{k=2}^{\infty}\Pr[X(a,c)>k]$, it follows that $\mathbb{E}[X(a,c)]=2+\frac{26}{27}=\frac{80}{27}$, which is strictly smaller than $\frac{10}{3}$.

In fact, the gap between the solution to Minimum Arrival and the solution to Best Policy can be arbitrarily large: Consider the graph consisting of vertices ${s}$ and ${y}$ and $n-2$ vertex disjoint paths of length 2 between ${s}$ and ${y}$. Assume also that, under the memoryless model, every edge incident to ${s}$ appears each day with probability $1$ and every edge incident to ${y}$ appears each day independently with probability $n^{-0.9}$. Similarly to the above example, the expected arrival time of a best policy for Alice is $h(s,y)=1+n^{0.9}$. On the other hand, the arrival time of the foremost journey from ${s}$ to ${y}$ will be equal to the first day after day 1 on which some edge incident to ${y}$ appears. But the time needed for the latter to happen follows the geometric distribution with success probability $1-(1-n^{-0.9})^{n-2}=1-o(1)$. Therefore, the expected arrival time of the foremost journey will be $\mathbb{E}[X(s,y)]=2+o(1)$, i.e. much smaller than $h(s,y)=1+n^{0.9}$.

As a final note, the expected arrival time $\mathbb{E}[X(s,y)]$ of the foremost ${s}$-${y}$ journey is always upper-bounded by the minimum among the expected values of the arrival times of all ${s}$-${y}$ journeys in the temporal graph. This is actually implied by a more general and well-known lemma in Probability Theory (Fatou’s lemma [16, p. 29]) which establishes that the expected value of the minimum among $n$ random variables is upper-bounded by the minimum among all the variables’ expectations.

3 Computing the expected minimum arrival time

3.1 Hardness of exact computation in the memoryless model

In this section we show that, even in the memoryless model, Minimum Arrival is #P-hard in both undirected graphs and directed acyclic graphs (DAGs). In the proof of the following theorem, the edges can be treated either as oriented, in which case we obtain the result for DAGs, or as non-oriented, in which case we obtain the result for undirected graphs.

Theorem 1**.**

Minimum Arrival* in the memoryless model is #P-hard.*

Proof.

To prove the theorem we will provide a reduction from the #P-complete problem #PP2DNF [35]. The latter problem is defined as follows. Let $X=\{x_{1},x_{2},\ldots,x_{n}\}$ and $Y=\{y_{1},y_{2},\ldots,y_{m}\}$ be two disjoint sets of Boolean variables. A positive, partitioned 2-DNF formula is a DNF formula of the form:

[TABLE]

for some $E\subseteq[n]\times[m]$. Given a positive, partitioned 2-DNF formula $\Phi$, the problem #PP2DNF asks for the number of truth assignments satisfying $\Phi$. Let $\Phi$ be an instance of #PP2DNF. We define $G$ to be a graph with the vertex set $\{{s},{y}\}\cup X\cup Y$ and the edge set $\{({s},x_{i})\leavevmode\nobreak\ |\leavevmode\nobreak\ x_{i}\in X\}\cup\{(x_{i},y_{j})\leavevmode\nobreak\ |\leavevmode\nobreak\ (i,j)\in E\}\cup\{(y_{i},{y})\leavevmode\nobreak\ |\leavevmode\nobreak\ y_{j}\in Y\}$, see Figure 1.

First we claim222This claim was provided by Antoine Amarilli (https://cstheory.stackexchange.com/q/42239). that the number $\psi$ of satisfying assignments of $\Phi$ is equal to the number of spanning subgraphs of $G$ which contain all the edges from $\{(x_{i},y_{j})\leavevmode\nobreak\ |\leavevmode\nobreak\ (i,j)\in E\}$ and have a simple path from ${s}$ to ${y}$ of length 3. To see the claim, for every subset $S\subseteq\{({s},x_{i})\leavevmode\nobreak\ |\leavevmode\nobreak\ x_{i}\in X\}\cup\{(y_{i},{y})\leavevmode\nobreak\ |\leavevmode\nobreak\ y_{j}\in Y\}$ of edges we define a truth assignment $\alpha$ that assigns $x_{i}=1$ iff $({s},x_{i})\in S$ and $y_{j}=1$ iff $(y_{j},{y})\in S$. Notice that every ${s}$-${y}$ path of length 3 in $G$ is of the form $({s},x_{i},y_{j},{y})$ for some $(i,j)\in E$. Therefore, if the subgraph spanned by $S$ contains a path $({s},x_{i},y_{j},{y})$, then $\alpha$ assigns 1 to both $x_{i}$ and $y_{j}$, and hence $\alpha$ satisfies $\Phi$. Conversely, given an assignment $\alpha$ satisfying $\Phi$, we define a subgraph of $G$ spanned by the edge set $\{({s},x_{i})\leavevmode\nobreak\ |\leavevmode\nobreak\ x_{i}\text{ is assigned 1 by }\alpha\}\cup\{(x_{i},y_{j})\leavevmode\nobreak\ |\leavevmode\nobreak\ (i,j)\in E\}\cup\{(y_{i},{y})\leavevmode\nobreak\ |\leavevmode\nobreak\ y_{j}\text{ is assigned 1 by }\alpha\}$. Since $\alpha$ is satisfying assignment, there exists $(i,j)\in E$ such $\alpha$ assigns 1 to both $x_{i}$ and $y_{j}$, and therefore the subgraph contains the ${s}$-${y}$ path $({s},x_{i},y_{j},{y})$ of length 3.

Now we define an instance of Minimum Arrival in the memoryless model as follows. Let $H$ be the graph obtained from $G$ by adding three new vertices $v_{1},v_{2},v_{3}$ and four new edges $({s},v_{1}),(v_{1},v_{2}),(v_{2},v_{3}),(v_{3},{y})$, which all together form a new ${s}$-${y}$ path of length 4. For every edge $e\in\{({s},x_{i})\leavevmode\nobreak\ |\leavevmode\nobreak\ x_{i}\in X\}\cup\{(y_{i},{y})\leavevmode\nobreak\ |\leavevmode\nobreak\ y_{j}\in Y\}$ we set $p_{e}=1/2$, and for any other edge $e$ of $H$ we set $p_{e}=1$. In this stochastic temporal graph the duration of a foremost journey from ${s}$ to ${y}$ is either 3, if for some $(i,j)\in E$ the edge $({s},x_{i})$ appears in time slot 1, and the edge $(y_{j},{y})$ appears in time slot 3, or 4 otherwise. In other words, the duration of a foremost ${s}$-${y}$ journey depends only on the subgraph of $G$ spanned by the edge set $R_{1}\subseteq\{({s},x_{i})\leavevmode\nobreak\ |\leavevmode\nobreak\ x_{i}\in X\}$ that appears in slot 1, and by the edge set $R_{3}\subseteq\{(y_{i},{y})\leavevmode\nobreak\ |\leavevmode\nobreak\ y_{j}\in Y\}$ that appears in slot 3. The duration is equal to 3 if and only if the subgraph of $G$ spanned by $R_{1}\cup\{(x_{i},y_{j})\leavevmode\nobreak\ |\leavevmode\nobreak\ (i,j)\in E\}\cup R_{3}$ has an ${s}$-${y}$ path of length 3. Since every edge in $R_{1}\cup R_{3}$ appears independently with probability $1/2$, it follows that the probability that this subgraph has a path of length 3 is equal to $p=\frac{\psi}{2^{n+m}}$. Consequently,

[TABLE]

and hence $\psi=2^{n+m}(4-\mathbb{E}[X({s},{y})])$. Therefore, knowing the expected duration $\mathbb{E}[X({s},{y})]$ of an ${s}$-${y}$ foremost journey, we can efficiently compute the number of satisfying assignments of $\Phi$, which proves that the computation of $\mathbb{E}[X({s},{y})]$ is #P-hard. ∎

Corollary 1**.**

For every $k\geq 0$, Minimum Arrival in the memory-$k$ model is #P-hard.

3.2 The FPTAS for the memoryless model on series-parallel graphs

3.2.1 The case of paths

In this section we will consider a stochastic temporal graph $\mathcal{P}^{(0)}=(P=(V,E),\{p_{e}\})$ with the underlying graph being a path $P=({s}=v_{0},v_{2},\ldots,v_{n}={y})$.

Lemma 1**.**

$\mathbb{E}[X_{\mathcal{P}^{(0)}}({s},{y})]=\sum_{e\in E}\frac{1}{p_{e}}$.

Proof.

Consider a stochastic temporal graph with a single edge $e$ which appears every day independently with probability $p_{e}$, and let $X_{e}$ be a random variable equal to the duration of the foremost journey from one of the endpoints of $e$ to the other. Then $X_{\mathcal{P}^{(0)}}({s},{y})=\sum_{e\in E}X_{e}$. Notice that $X_{e}$ is a geometric random variable with probability mass function $\Pr[X_{e}=i]=(1-p_{e})^{i-1}p_{e}$ for $i=1,2,3,...$, and expectation $\mathbb{E}[X_{e}]=\frac{1}{p_{e}}$. Therefore $\mathbb{E}[X_{\mathcal{P}^{(0)}}({s},{y})]=\sum_{e\in E}\mathbb{E}[X_{e}]=\sum_{e\in E}\frac{1}{p_{e}}$. ∎

Let us denote by $\mu$ the expectation $\mu\stackrel{{\scriptstyle\text{def}}}{{=}}\mathbb{E}[X_{\mathcal{P}^{(0)}}({s},{y})]=\sum_{e\in E}\frac{1}{p_{e}}$. Note that

[TABLE]

In the remainder of this section we will show that the first $O(\mu\ln\mu)$ terms of sum (1) already give a very good approximation of $\mu$. In our analysis we will use the following bound.

Theorem 2** ([25]).**

Let $X=\sum_{i=1}^{n}X_{i}$, where $n\geq 1$ and $X_{i}$, $i=1,\ldots,n,$ are independent geometric random variables with parameters $p_{1},p_{2},\ldots,p_{n}\in(0,1]$, respectively. Let $\mu=\mathbb{E}[X]=\sum_{i=1}^{n}\frac{1}{p_{i}}$. Then for any $\lambda\geq 1$,

[TABLE]

Lemma 2**.**

Let $\varepsilon$ be a number such that $0<\varepsilon\leq 1$. Then

[TABLE]

for every $\tau\geq\mu\left(\ln\frac{\mu}{\varepsilon}+1\right)$, where $\mu=\mathbb{E}[X_{\mathcal{P}^{(0)}}({s},{y})]$.

Proof.

The equality in (2) follows from (1). In the rest of the proof we show the inequality. Since $\tau\geq\mu\left(\ln\frac{\mu}{\varepsilon}+1\right)\geq\mu$, using Theorem 2 we have

[TABLE]

where we used the inequality $e^{x}\geq 1+x+x^{2}/2$ which holds for every $x\geq 0$. ∎

3.2.2 A general FPTAS approach

While deriving analytically and computing efficiently the exact solution of Minimum Arrival in a path is an easy task (cf. Lemma 1), it does not seem to be trivial for a slight generalization of paths, called parallel compositions of paths. A parallel composition of paths is the graph obtained from a collection of disjoint paths $P_{1},P_{2},\ldots,P_{\ell}$ with end vertices ${s}_{i},{y}_{i}$, $i=1,\ldots,\ell$, respectively, by identifying the vertices ${s}_{1},{s}_{2},\ldots,{s}_{\ell}$ in a single vertex ${s}$, and by identifying the vertices ${y}_{1},{y}_{2},\ldots,{y}_{\ell}$ in a single vertex ${y}$.

It is not clear whether there exists an efficient procedure for computing the expected arrival time from ${s}$ to ${y}$ in a parallel composition of paths, even if the parallel paths are of equal length and all the probabilities of edge appearance are the same. In this section we present a general approach for developing $\varepsilon$-additive approximation algorithms333A feasible solution is $\varepsilon$-additive approximate if it is within $\varepsilon$ additive factor from the optimal value. An algorithm is called an $\varepsilon$-additive approximation algorithm if it returns an $\varepsilon$-additive approximate solution for any instance. for computing the expected arrival time of a foremost journey in special classes of stochastic temporal graphs. In Section 3.2.3 we apply this approach to develop an efficient $\varepsilon$-additive approximation algorithm for the problem on the class of stochastic temporal graphs with underlying graphs being series-parallel graphs, which generalize parallel compositions of paths and graphs, in which all simple ${s}$-${y}$ paths are of the same length.

Throughout the section we denote by $\mathcal{G}^{(0)}=(G=(V,E),\{p_{e}\})$ a memoryless stochastic temporal graph with $n$ vertices and $m$ edges, and by ${s},{y}\in V$ two distinct vertices in $G$. Furthermore, we denote by $H=(V,E,w)$ the weighted graph obtained from the underlying graph $G$ by assigning to every edge $e\in E$ the weight $w(e)=\frac{1}{p_{e}}$.

Definition 6**.**

Let $\mathcal{G}^{(0)}$ be a memoryless stochastic temporal graph, where $G$ is the underlying graph. A stochastic temporal subgraph $\mathcal{H}^{(0)}$ of $\mathcal{G}^{(0)}$ is a stochastic temporal graph which has a subgraph $H\subseteq G$ as an underlying graph and inherits all edge appearance probabilities from $\mathcal{G}^{(0)}$.

Observation 1**.**

Let $\mathcal{H}^{(0)}$ be a stochastic temporal subgraph of the stochastic temporal graph $\mathcal{G}^{(0)}$. Then for every natural number $i$ we have $\Pr[X_{\mathcal{G}^{(0)}}({s},{y})\geq i]\leq\Pr[X_{\mathcal{H}^{(0)}}({s},{y})\geq i]$, and hence $\mathbb{E}[X_{\mathcal{G}^{(0)}}({s},{y})]\leq\mathbb{E}[X_{\mathcal{H}^{(0)}}({s},{y})]$.

The following lemma is a direct consequence of Observation 1 and Lemma 1.

Lemma 3**.**

Let $w^{*}$ be the minimum weight of an ${s}$-${y}$ path in $H$. Then $\mathbb{E}[X_{\mathcal{G}^{(0)}}({s},{y})]\leq w^{*}$.

Theorem 3**.**

Let $c\in\mathbb{N}$ and $\varepsilon\in(0,1]$. Let $p_{e}\geq\frac{1}{n^{c}}$ for every $e\in E$ and suppose that there exists an algorithm $A$ that computes in time $O\left(f(\ell,n,m)\right)$ the probabilities $\Pr[X_{\mathcal{G}^{(0)}}({s},{y})\geq i]$ for all $i=1,\ldots,\ell$. Then there exists an algorithm $B$ that approximates $\mathbb{E}[X_{\mathcal{G}^{(0)}}({s},{y})]$ within the additive factor of $\varepsilon$ in time

[TABLE]

Consequently, if $f(\ell,n,m)$ is a polynomial in variables $\ell,n$, and $m$, then $B$ is an FPTAS on the instance $(\mathcal{G}^{(0)},{s},{y})$.

Proof.

Let $P=(s=v_{0},v_{1},\ldots,v_{r}=y)$ be a minimum weight ${s}$-${y}$ path in $H$, and let $\mathcal{P}^{(0)}$ be the stochastic temporal subgraph of $\mathcal{G}^{(0)}$ restricted to the vertices of $P$. For convenience, let us denote $e_{i}=v_{i-1}v_{i}$ for every $i=1,\ldots,r$. Then, by definition and Lemma 1, the weight $w^{*}$ of $P$ is equal to $\sum_{i=1}^{r}\frac{1}{p_{e_{i}}}=\mathbb{E}[X_{\mathcal{P}^{(0)}}({s},{y})]$. Let $\tau:=w^{*}\left(\ln\frac{w^{*}}{\varepsilon}+1\right)$. Then, by Observation 1 and Lemma 2, we have that

[TABLE]

and hence

[TABLE]

that is, $\sum_{i=1}^{\tau}\Pr[X_{\mathcal{G}^{(0)}}({s},{y})\geq i]$ approximates $\mathbb{E}[X_{\mathcal{G}^{(0)}}({s},{y})]$ within the additive factor of $\varepsilon$.

Now we define the desired algorithm $B$ as follows:

Construct the graph $H$ and compute the minimum weight $w^{*}$ of an ${s}$-${y}$ path in $H$ using Dijkstra’s algorithm. 2. 2.

Using algorithm $A$, compute the probabilities $\Pr[X_{\mathcal{G}^{(0)}}({s},{y})\geq i]$ for every $i=1,\ldots,\tau$, where $\tau=w^{*}\left(\ln\frac{w^{*}}{\varepsilon}+1\right)$. 3. 3.

Output $\sum_{i=1}^{\tau}\Pr[X_{\mathcal{G}^{(0)}}({s},{y})\geq i]$.

The above discussion implies that algorithm $B$ correctly computes the declared approximation of $\mathbb{E}[X_{\mathcal{G}^{(0)}}({s},{y})]$. It remains to justify the time complexity. First, Dijkstra’s algorithm can be implemented to work in time $O(n\ln n+m)$ [21]. Second, the assumption on $p_{e}$’s implies that $w^{*}=O(n^{c+1})$, and hence $\tau=w^{*}\left(\ln\frac{w^{*}}{\varepsilon}+1\right)=O\left(n^{c+1}\ln\frac{n}{\varepsilon}\right)$. Therefore the assumption of the theorem implies that the last two steps of the algorithm run in time O\Big{(}f\big{(}n^{c+1}\ln\frac{n}{\varepsilon},n,m\big{)}\Big{)}, which in turn implies the complexity bound and completes the proof. ∎

3.2.3 The FPTAS for stochastic temporal series-parallel graphs

In the present section we use the approach from Section 3.2.2 to derive a polynomial-time approximation scheme for stochastic temporal series-parallel graphs. According to Theorem 3, the development of such an algorithm reduces to the design of an efficient procedure of computing probabilities of the form $\Pr[X_{\mathcal{G}^{(0)}}({s},{y})\geq i]$, which is the main goal of this section.

Let $G$ be a graph and ${s}$ and ${y}$ be two distinct vertices in $G$. The triple $(G,{s},{y})$ is a two-terminal series-parallel graph, with terminals ${s}$ and ${y}$, if $G$ can be constructed by a sequence of the following two operations starting from a set of copies of a single-edge two-terminal series-parallel graph $(K_{2},a,b)$.

Parallel composition: Given a pair of two-terminal series-parallel graphs $(H_{1},{s}_{1},{y}_{1})$ and $(H_{2},{s}_{2},{y}_{2})$, form a new two-terminal series-parallel graph $(G,{s},{y})$ by identifying ${s}={s}_{1}={s}_{2}$ and ${y}={y}_{1}={y}_{2}$. 2. 2.

Series composition: Given a pair of two-terminal series-parallel graphs $(H_{1},{s}_{1},{y}_{1})$ and $(H_{2},{s}_{2},{y}_{2})$, form a new two-terminal series-parallel graph $(G,{s},{y})$ by identifying ${s}={s}_{1}$, ${y}_{1}={s}_{2}$, and ${y}={y}_{2}$.

Finally, a graph $G$ is called series-parallel if $(G,{s},{y})$ is a two-terminal series-parallel graph for some pair of distinct vertices ${s}$ and ${y}$ of $G$.

The sequence of parallel and series compositions leading to a two-terminal series-parallel graph $(G=(V,E),{s},{y})$ can be conveniently represented by a decomposition tree. A binary tree $T=(V_{T},E_{T})$ with a labeling function $\sigma:V_{T}\rightarrow\{\texttt{s,p}\}\cup E\times\{0,1\}$ is called a decomposition tree of a two-terminal series-parallel graph $(G,{s},{y})$ if and only if the leaves of $T$ are labeled with elements of $E\times\{0,1\}$ such that every $e\in E$ appears in exactly one label, internal nodes are labeled with p or s, and $G$ can be generated recursively using $T$ as follows: If $T$ is a single node $v$ with $\sigma(v)=(e,\alpha)$, then $G$ consists of the single edge $e$ with the source being the vertex with the smallest ID, if $\alpha=0$, and with the source being the vertex with the largest ID, if $\alpha=1$. Otherwise, let $T_{1}$ (resp. $T_{2}$) be the right (resp. left) subtree of $T$ and $(H_{1},{s}_{1},{y}_{1})$ and $(H_{2},{s}_{2},{y}_{2})$ be two-terminal series-parallel graphs with decomposition trees $T_{1}$ and $T_{2}$: if $\sigma(v)=\texttt{p}$ (resp. s) then $G$ is the parallel (resp. series) composition of $(H_{1},{s}_{1},{y}_{1})$ and $(H_{2},{s}_{2},{y}_{2})$.

We will make use of tree decompositions of series-parallel graphs in our algorithm. It is known that a tree decomposition of a series-parallel graph can be constructed in linear time.

Theorem 4** ([39]).**

Given a two-terminal series-parallel graph with $n$ vertices and $m$ edges, its tree decomposition can be computed in time $O(n+m)$.

Let $\mathcal{G}^{(0)}=(G=(V,E),\{p_{e}\})$ be a stochastic temporal graph with the underlying graph $G$ being series-parallel. Let also ${s},{y}\in V$ be two distinct vertices in $G$ such that $(G,{s},{y})$ is a two-terminal series-parallel graph. We will present a dynamic programming algorithm which, for a given natural number $\ell$, computes the set of probabilities:

[TABLE]

For convenience, the algorithm will also support the set of probabilities:

[TABLE]

Notice that having computed one of the sets of probabilities, the other set can be computed in $O(\ell^{2})$ time.

The algorithm is based on the following recursive equations. Since $(G,{s},{y})$ is a two-terminal series-parallel graph, it is either a single-edge graph, or can be obtained from smaller two-terminal series-parallel graphs $(H_{1},{s}_{1},{y}_{1})$, $(H_{2},{s}_{2},{y}_{2})$ via one of the two composition operations.

In the case of a single-edge graph we have for every $i\in[\ell-1]$ that:

[TABLE]

where $p$ is the probability of appearance of the unique edge of the graph. 2. 2.

In the case of parallel composition we have for every $i\in[\ell]$ that:

[TABLE]

where $\mathcal{H}^{(0)}_{1}$ and $\mathcal{H}^{(0)}_{2}$ are the stochastic temporal subgraphs of $\mathcal{G}^{(0)}$ restricted to the vertices of $H_{1}$ and $H_{2}$, respectively. 3. 3.

In the case of series composition, we have for every $i\in[\ell-1]$ that:

[TABLE]

Theorem 5**.**

Given a stochastic temporal series-parallel graph $\mathcal{G}^{(0)}=(G=(V,E),\{p_{e}\})$, two vertices ${s},{y}\in V$ such that $(G,{s},{y})$ is a two-terminal series-parallel graph, and a natural number $\ell$, Algorithm 1 correctly computes the probabilities $\{\Pr[X_{\mathcal{G}^{(0)}}({s},{y})\geq i]:i\in[\ell]\}$ and $\{\Pr[X_{\mathcal{G}^{(0)}}({s},{y})=i]:i\in[\ell-1]\}$ in time $O(m\ell^{2})$, where $m=|E|$.

Proof.

We start with the analysis of the correctness of the algorithm. First, if the underlying graph of the input stochastic temporal graph is a single-edge graph, then the algorithm computes the required sets of probabilities in lines 2-4 using equations (5). Second, if the underlying graph is not a single-edge graph, then, by definition, $(G,{s},{y})$ is either the parallel or the series composition of two two-terminal series-parallel graphs $(H_{1},{s}_{1},{y}_{1})$ and $(H_{2},{s}_{2},{y}_{2})$ whose decomposition trees are the subtrees $T_{H_{1}}$ and $T_{H_{2}}$ of $T_{G}$ rooted at the children of the root of $T_{G}$. In the case of parallel composition, the algorithm computes the sets of probabilities in lines 10-12 using equations (6). In the case of series composition, the algorithm computes the sets of probabilities in lines 14-16 using equations (7). In both cases, the computation of the probabilities uses only the corresponding sets of probabilities for the stochastic temporal subgraphs $\mathcal{H}^{(0)}_{1}$ and $\mathcal{H}^{(0)}_{2}$, which are computed recursively in lines 7 and 8, respectively.

In order to analyze the complexity of Algorithm 1, we observe that for every node of the decomposition tree of the underlying graph the algorithm makes exactly one recursive call. In each of the calls, the algorithm executes either lines 2-4, or lines 10-12, or lines 14-16. It is easy to check that each of these sets of lines performs $O(\ell^{2})$ operations of addition or multiplication. Since $T_{G}$ is a binary tree and has exactly $m$ leaves, in total $T_{G}$ has $2m-1$ nodes, and therefore the total time complexity of Algorithm 1 is $O(m\ell^{2})$. ∎

Finally we present an FPTAS for the expected arrival time of a foremost ${s}$-${y}$ journey in a stochastic temporal series-parallel graph.

The following theorem follows from the proof of Theorem 3 and Theorem 5.

Theorem 6**.**

Algorithm 2 is correct and works in time O\Big{(}m\cdot n^{2c+2}\ln^{2}\frac{n}{\varepsilon}\Big{)}, where $n$ and $m$ are the number of vertices and the number of edges in the underlying graph, respectively.

3.3 The FPRAS for general graphs in the memory-$k$ model, $k\geq 0$

In this section, we present our FPRAS for Minimum Arrival in the memory-$k$ model, for every $k\geq 0$, under the assumption that every edge appearance probability is lower bounded by $\frac{1}{n^{c}}$ for some $c\geq 1$.

Lemma 4**.**

Let $\mathcal{G}^{(k)}=(G=(V,E),\{p_{e}(H_{e}^{(k)})\})$ be a memory-$k$ stochastic temporal graph, and let $Z$ be a geometric random variable with success probability $\frac{1}{n^{c}}$. Then we have

$\Pr\left[X({s},{y})\geq a\right]\ \leq\ \Pr\left[(n-1)Z\geq a\right]$, for every $a\in\mathbb{N}$, and 2. 2.

$\mathbb{E}[X({s},{y})]\ \leq\ \mathbb{E}[(n-1)Z]\ \leq\ n^{c+1}$.

Proof.

Let $e\in E$ be an arbitrary edge in the underlying graph and let $t\in\mathbb{N}$ be an arbitrary time point. Let $X_{e}^{t}$ be the random variable equal to the shortest time required to move from one of the end vertices of $e$ to the other starting at time $t$. Then for any $a\in\mathbb{N}$ we have that

[TABLE]

where the inequality follows from our assumed lower bound on edge appearance probabilities. Now, since an edge and time point were picked arbitrarily and any $s$-$y$ foremost journey traverses at most $n-1$ edges, we conclude (1), that is, $\Pr\left[X({s},{y})\geq a\right]\leq\Pr\left[(n-1)Z\geq a\right]$, for every $a\in\mathbb{N}$. Now, using (1) we establish (2) as follows:

[TABLE]

∎

In the following theorem we provide our FPRAS for Minimum Arrival.

Theorem 7**.**

Let $\varepsilon\in(0,1)$ and let $\mathcal{G}^{(k)}$ be a memory-$k$ stochastic temporal graph with two designated vertices ${s},{y}$. Furthermore let every edge appearance probability be at least $\frac{1}{n^{c}}$ for some $c\geq 1$. Then Minimum Arrival admits an FPRAS which runs in $O\left(m\frac{n^{5c+8}}{\varepsilon^{4}}\cdot\log(\frac{n}{\varepsilon})\right)$ time with probability of success at least $1-\frac{2}{n}$.

Proof.

Let $\mathcal{G}^{(k)}$ be a stochastic temporal graph with two designated vertices ${s},{y}$. Furthermore let $X$, as before, be the arrival time of a foremost ${s}$-${y}$ journey. We will estimate the expectation $\mathbb{E}(X)$ via an unbiased estimator approach. We perform $r$ times independently the following experiment Exp; for now let us assume an arbitrary value for $r$, to be chosen precisely later.

The probability that Exp fails to connect ${s}$ to ${y}$ via a journey is equal to the probability that ${s}$ is not connected to ${y}$ until time $t^{\prime}$. Therefore, Lemma 4 implies that the time to connect ${s}$ to ${y}$ exceeds the expectation $\mathbb{E}(X)$ of $X$ by a multiplicative factor of at least $rn$. By Markov’s inequality, this probability of failure is at most $\frac{1}{rn}$. For now, we proceed the analysis of the algorithm assuming that all experiments succeed, and we will take the probability of failure of some experiment(s) into account later on.

Let $X_{i}$ be the random variable returned by the $i$th execution of the experiment Exp, where $i=1,2,\ldots,r$, and let $\widetilde{X}=\frac{1}{r}(X_{1}+\ldots+X_{r})$ be the estimator for $X$. Note that $\mathbb{E}[\widetilde{X}]=\mathbb{E}[X]$, meaning that $\widetilde{X}$ is an unbiased estimator for $X$. Thus, it follows by Chebyshev’s inequality that, for every $\varepsilon\in(0,1)$:

[TABLE]

It holds that $\sigma(\widetilde{X})=\frac{\sigma(X)}{\sqrt{r}}$ (see [40, p. 297]), hence (8) becomes:

[TABLE]

We will now show that $\sigma(X)$ is upper bounded by a polynomial in $n$. Indeed, we have:

[TABLE]

Now, consider any ${s}$-${y}$ journey in $\mathcal{G}^{(k)}$ and let $\ell$ be the number of its edges. Let $Y_{1}$ be the number of time steps needed until we cross the $1$st edge of this journey, starting at time [math]. Also let $Y_{i}$ be the number of time steps that are needed, after we cross the $(i-1)$th edge, until we cross the $i$th edge of this journey, $i=2,\ldots,\ell$. Then $X\leq Y_{1}+\ldots+Y_{\ell}$, and thus

[TABLE]

Note that each $Y_{i}$ is dominated by a geometric random variable $Z$ with success probability $\frac{1}{n^{c}}$. Therefore, it follows by the properties of geometric random variables that $\mathbb{E}(Y^{2}_{i})\leq 2n^{2c}$. Thus $\mathbb{E}[X^{2}]\leq n(\mathbb{E}[Y_{1}^{2}]+\ldots+\mathbb{E}[Y_{\ell}^{2}])\leq 2n^{2c+1}\ell\leq 2n^{2c+2}$. So, (10) becomes:

[TABLE]

Going back to (9), it becomes:

[TABLE]

So, for a number $r=2\frac{n^{2c+3}}{\varepsilon^{2}}$ of experiments, we get:

[TABLE]

meaning that performing $r$ independent times the experiment Exp results in polynomial time in a solution $\widetilde{X}$ that is within a factor $(1\pm\varepsilon)$ of the optimal with probability at least $1-\frac{1}{n}$. Let us call the probability that $\widetilde{X}$ is far from the optimal solution “probability of failure of the estimator”. Recall that there is also a chance of failure in our algorithm if any of the $r$ experiments fails. Therefore, the probability of failure of our FPRAS is:

[TABLE]

We execute the experiment Exp for $r=2\frac{n^{2c+3}}{\varepsilon^{2}}$ times. Each execution of the experiment runs in total for $t^{\prime}=2\frac{n^{3c+5}}{\varepsilon^{2}}$ time; during this time we get at most $t^{\prime}m$ time-edges, i.e. the algorithm of [3] runs in $O(t^{\prime}m\log(t^{\prime}m))=O(m\frac{n^{3c+5}}{\varepsilon^{2}}\cdot\log(\frac{n}{\varepsilon}))$ time. Thus the total running time is $O(m\frac{n^{5c+8}}{\varepsilon^{4}}\cdot\log(\frac{n}{\varepsilon}))$. ∎

4 Computing the expected arrival time of a best policy

In this section we investigate the computational complexity of our second problem, namely Best Policy.

4.1 A polynomial-time algorithm for the memoryless model

In this section we focus on the memoryless model and we derive a polynomial-time dynamic-programming algorithm for Best Policy. We define for every vertex $v$ the expected arrival time $h(v,{y})\stackrel{{\scriptstyle\text{def}}}{{=}}\mathbb{E}_{\mathcal{G}^{(0)}}[Y(v,{y})]$ of the $v$-${y}$ journey suggested to Alice by a best policy (i.e. when Alice starts her journey at vertex $v$). For simplicity of presentation, throughout Section 4.1 we write $h(v)\stackrel{{\scriptstyle\text{def}}}{{=}}h(v,{y})$.

Assume for now that for all $v\in V$, the value $h(v)$ is given; let $v_{1}={y},v_{2},\ldots,v_{n}$ be an ordering of vertices of $V$ in non-decreasing values of $h$ (ties broken arbitrarily), namely $h(v_{1})\leq h(v_{2})\leq\cdots\leq h(v_{n})$. Clearly, $v_{1}=y$ and $h(v_{1})=h(y)=0$.

Let $s_{t}$ be the vertex that Alice occupied at time $t$ and recall that $\Gamma_{G_{t}}(v)$ is the neighborhood of vertex $v$ in the snapshot $G_{t}$, for all $v\in V$ and all $t\in\mathbb{N}$. Notice that, the best strategy of Alice at time $t+1$ is to look at all neighboring vertices of $s_{t}$ in $G_{t+1}$ and find one with minimum $h$-value, namely a vertex $u\in\arg\min\{h(v):v\in\Gamma_{G_{t+1}}(s_{t})\}$. If $h(u)\geq h(s_{t})$, then Alice has no incentive to change vertex and thus $s_{t+1}=s_{t}$. Otherwise, if $h(u)<h(s_{t})$, then $s_{t+1}=u$.

Therefore, to find the best choice for Alice, it suffices to find the values $h(v),v\in V$. In view of the above, if Alice is on vertex $v_{i}$ at time 0 (i.e. she is on the $i$-th best vertex in terms of closeness to $y$), she will move to the $j$-th best (with $j<i$) only if an edge appears between $v_{i}$ and $v_{j}$ in the next step, and no edge to a vertex better than $v_{j}$ appears (i.e. no edge between $v_{i}$ and $v_{\ell},\leavevmode\nobreak\ 1\leq\ell\leq j-1$). This happens with probability $Q_{i,j}=p_{\{v_{i},v_{j}\}}\prod_{\ell=1}^{j-1}(1-p_{\{v_{i},v_{\ell}\}})$, where $\{v_{i},v_{\ell}\}$ denotes the (undirected) edge between $v_{i}$ and $v_{\ell}$. Additionally, with probability $Q_{i}=\prod_{\ell=1}^{i-1}(1-p_{\{v_{i},v_{\ell}\}})$ no edge to a vertex better than $v_{i}$ will appear, in which case Alice will stay on $v_{i}$. Therefore $h(v_{i})$ can be recurrently computed by $h(v_{i})=\sum_{j=1}^{i-1}Q_{i,j}h(v_{j})+Q_{i}h(v_{i})+1$, or equivalently:

[TABLE]

with initial condition $h(v_{1})=0$. Indeed, the above equation follows by observing that the expected length of the foremost journey to $y$ when Alice is on $v_{i}$ is equal to $1+h(v_{1})$ with probability $Q_{i,1}$ (which is the probability that an edge between $v_{i}$ and $v_{1}=y$ exists), plus $1+h(v_{2})$ with probability $Q_{i,2}$ (which is the probability that and edge between $v_{i}$ and the second best vertex $v_{2}$ exists, but there is no edge between $v_{i}$ and $v_{1}$), and so on. In general, the above recurrence states that there is no incentive to visit vertices with larger index and also Alice will visit the smallest index vertex $v_{j}$ for which the edge $\{v_{i},v_{j}\}$ is present (otherwise, if no such edge exists, she will stay on $v_{i}$). Using the above recurrence, we can compute all values of $h(v_{i})$ by the following bottom-up dynamic programming algorithm444To avoid trivialities, we assume that the graph induced by the set of edges with non-zero probabilities is connected; in particular, the set of probabilities can be used to model any connected underlying graph $G$. We also assume that the elements of the list $L$ can be accessed using their index, i.e. $L_{i}$ is the $i$-th element of the list.:

Algorithm 4 can be efficiently implemented to run in $O(n^{2})$ time and space. This can be achieved by carefully storing intermediate sums and products in step 4; indeed, at step $i+1$ the only new term in the numerator is $p_{\{v,L_{i}\}}\prod_{\ell=1}^{i-1}(1-p_{\{v,L_{\ell}\}})h(L_{i})$, and in the product $\prod_{\ell=1}^{i}(1-p_{\{v,L_{\ell}\}})$ which appears in the denominator the only new factor is $1-p_{\{v,L_{i}\}}$. Concluding, we have the following theorem:

Theorem 8**.**

Best Policy* can be optimally computed in the memoryless model in $O(n^{2})$ time and space.*

4.2 Hardness of computation for the memory-$k$ model, $k\geq 3$

We now show that Best Policy is #P-hard for memory-3 stochastic temporal graphs on directed acyclic graphs, and consequently also for memory $k\geq 3$.

Theorem 9**.**

When the underlying graph is a Directed Acyclic Graph (DAG), it is #P-hard to compute the expected arrival time of the best policy journey in the memory-3 model.

Proof.

We will provide a reduction from the counting problem #PP2DNF which is known to be #P-hard [35]. Recall that this problem takes as input a DNF formula $\Phi=\bigvee_{(i,j)\in E}x_{i}y_{j}$ on the sets of variables $X=\{x_{1},\ldots,x_{n}\}$ and $Y=\{y_{1},\ldots,y_{m}\}$, for some $E\subseteq[n]\times[m]$, and the task is to compute the number $\psi$ of truth assignments that satisfy $\Phi$. Similarly to our reduction in the proof of Theorem 1, we create a directed acyclic graph (DAG) $H$ as follows. First, $H$ has one vertex for each of the variables in $X\cup Y$; then we add two distinct vertices ${s},{y}$ and one other vertex $v$. For every vertex $x_{i}\in X$ and every vertex $y_{i}\in Y$ we add the directed edges $({s},x_{i})$ and $(y_{j},{y})$. Furthermore we add the edge $(x_{i},y_{j})$ whenever $x_{i}y_{j}$ is a clause in $\Phi$. Finally we add the edges $({s},v)$ and $(v,{y})$. The construction of $H$ is illustrated in Figure 2.

Denote by $M=5\cdot 2^{n+m}$, and assume that $2^{n+m}\geq 3$ in order to avoid trivialities. All edges $(x_{i},y_{j})$ appear constantly in $H$, i.e. they appear at every time step $i\geq 1$ in a memoryless fashion with probability $1$. Both edges $({s},v)$ and $(v,{y})$ also appear in a memoryless fashion, each of them with probability $\frac{2}{M}$ at every step $i\geq 1$. Moreover, each of the edges $({s},x_{i})$ and $(y_{j},{y})$ appears at each step $i\geq 1$ according to the following table of memory $3$. This table has four columns and eight rows. Each column is labeled with the sequence of consecutive time steps $i-3,i-2,i-1$, and $i$. Each row corresponds to a different triple of appearances of each of the edges in $\{({s},x_{i}),(y_{j},{y}):x\in X,y\in Y\}$ at the time steps $i-3,i-2,i-1$ (here $1$ means “edge exists” and [math] means “edge does not exist”). At the end of each row there is a pair of numbers $(p,1-p)$ which denotes that, with the particular history of memory $3$, at time step $i$ the edge appears with probability $p$ and it does not appear with probability $1-p$. For simplicity of notation, in the column of time step $i$, we write “0” and “1” to denote the entries $(0,1)$ and $(1,0)$, respectively.

[TABLE]

To complete the description of our memory-3 instance, we specify that, in the fictitious initialization snapshots $G_{-2},G_{-1},G_{0}$, each of the edges $({s},x_{i})$ and $(y_{j},{y})$ appears with probability [math], [math], and $1$, respectively, i.e. according to the first row of the above table.

The intuition of this table for the edges $({s},x_{i})$ and $(y_{j},{y})$ is as follows. In the snapshot $G_{1}$, none of these edges appears (see the first line of the table). Then, to determine whether each of these edges appears at time step $2$ (see the second row of the table), we need to toss an unbiased coin which with probability $\frac{1}{2}$ outputs “appear” and with probability $\frac{1}{2}$ outputs “does not appear”. Once this coin has been tossed at time step $2$, the status of the edge does not change any more in any subsequent time step $i\geq 3$. That is, if one of the edges $({s},x_{i})$ and $(y_{j},{y})$ appears (resp. does not appear) at time $2$, then it appears (resp. does not appear) at all times $i\geq 3$ too. This is easy to be verified by observing the rows $3$-$7$ of the table. Note that the last row of the table is included only for the sake of completeness, as it does not affect the appearance of any edge of $H$ at any time step $i$.

Let $\ell$ be the expected ${s}$-${y}$ arrival time of the best policy in the memory-3 model. Note that, from the above construction of the temporal graph instance, each of the edges $({s},x_{i})$ and $(y_{j},{y})$ appears with probability $\frac{1}{2}$ at all steps $i\geq 2$, while it does not appear at any step $i\geq 2$ with probability $\frac{1}{2}$. Therefore, the probability that there exists a directed temporal path $({s},x_{i},y_{j},{y})$ is equal to $g=\frac{\psi}{2^{n+m}}$, where $\psi$ is the number of satisfying truth assignments of the DNF formula $\Phi$. That is, with probability $1-g$, there exists no such temporal path from ${s}$ to ${y}$ with 3 edges through some vertices $x_{i}$ and $y_{j}$. Furthermore, the expected ${s}$-${y}$ arrival time through the edges $({s},v)$ and $(v,{y})$ is equal to $\frac{M}{2}+\frac{M}{2}=M$. Therefore, since with probability $1-g$ any policy (also the best one) needs to travel from ${s}$ to ${y}$ through vertex $v$, it follows that $\ell\geq M(1-g)$.

We now define the following policy: at time step $1$ do nothing and just wait for the outcome of the random coin tosses which occur at time step $2$. Subsequently, at time step $2$ do the following: if there exists a directed temporal path $({s},x_{i},y_{j},{y})$ then follow it, starting at time step $2$; otherwise follow the temporal path $({s},v,{y})$ which has an expected travel time $\frac{M}{2}+\frac{M}{2}=M$. The expected arrival time of this particular policy is equal to $1+3g+M(1-g)$, and thus it follows that $\ell\leq 1+3g+M(1-g)$. Summarizing, we have:

[TABLE]

The first inequality can be written as $2^{n+m}-\frac{\ell}{5}\leq\psi$, while the second one can be written as $\left(1-\frac{3}{5\cdot 2^{n+m}}\right)\psi\leq 2^{n+m}-\frac{\ell}{5}+\frac{1}{5}$. Therefore:

[TABLE]

and thus

[TABLE]

Thus, knowing the expected value $\ell$ for the best policy we can derive the exact integer value for $\psi$ in the counting problem #PP2DNF. This completes the #P-hardness reduction. ∎

4.3 An exact algorithm for the memory-$k$ model, $k\geq 1$

In this section we present a doubly exponential-time exact algorithm for computing the best policy for Alice in the memory-$k$ model, where $k\geq 1$. We first give a Markov Decision Process (MDP) formulation of our problem under the memory-$k$ model that will be useful for the presentation of our results within the general MDP framework.

4.3.1 An MDP formulation

We follow the notation from chapter 5.4 of [32] to give an MDP formulation. Let $k\geq 1$ be a fixed integer corresponding to the memory of the model. We denote by ${\cal I}=V\times(2^{G})^{k}$ the set of states, where $2^{G}$ denotes the set of subgraphs of the underlying graph $G$. In particular, each state $(v,H^{(k)})\in{\cal I}$ consists of a vertex $v$ which corresponds to the vertex Alice is on and a sequence of $k$ graphs $H^{(k)}$ corresponding to the $k$ most recent snapshots. For any $t\geq 0$, we will say that $H^{(k)}_{t}\stackrel{{\scriptstyle\text{def}}}{{=}}\left(G_{t-k+1},G_{t-k+2},\ldots,G_{t-1},G_{t}\right)$ occurred at time $t$, if the snapshots at times $t-k+1,t-k+2,\ldots,t-1,t$ are $G_{t-k+1},G_{t-k+2},\ldots,G_{t-1},G_{t}$, respectively. The set of actions for Alice is the set ${\cal A}=V$. A stationary policy for Alice is a function $f:{\cal I}\to{\cal A}$ and determines a probability law $\Pr^{f}$ for a Markov chain $(X_{t})_{t\geq 0}$ with values in ${\cal I}$ as follows:

(i)

Assuming that at time 0 Alice starts from vertex $s$ and the initial sequence of $k$ snapshots is $H^{(k)}_{0}=\left(G_{-k+1},G_{-k+2},\ldots,G_{-1},G_{0}\right)$, the initial distribution is given by $\Pr^{f}\left[X_{0}=(s,H^{(k)}_{0})\right]=1$, and $\Pr^{f}\left[X_{0}=(v,H^{(k)})\right]=0$ if $v\neq s$ or $H^{(k)}\neq H^{(k)}_{0}$.

(ii)

For any $t\geq 0$,

[TABLE]

Without loss of generality, we will assume that every policy $f$ is legitimate in the sense that the following conditions hold:

A.

$f(v_{t},H^{(k)}_{t+1})=v_{t+1}$ only if $(v_{t},v_{t+1})\in E(G_{t+1})$, i.e. Alice may visit $v_{t+1}$ in the next step only if $G_{t+1}$ has an edge that connects $v_{t}$ (the vertex she is currently on) and $v_{t+1}$ (the vertex she wants to go to).

B.

Recalling that the goal of Alice is to reach $y$, we assume that $f(y,H^{(k)})=y$, for any $H^{(k)}$, i.e. Alice will never leave her target vertex once she reaches it.

For simplicity, we will denote by $a_{t}$ Alice’s $t$-th action (vertex choice). In particular, $a_{0}=s$ and inductively $a_{t+1}=f(a_{t},H^{(k)}_{t+1})$, for any $t\geq 0$. Furthermore, let \mu(G_{t+1}|H^{(k)}_{t})\stackrel{{\scriptstyle\text{def}}}{{=}}\Pr[\textrm{G_{t+1}$occurs at$t+1}|\textrm{H^{(k)}_{t}$occurred at$t}].

To complete the specification of the Markov Decision Process, we assume that constant cost $c((v,H^{(k)}),a)=1$ is incurred when action $a$ is chosen in state $(v,H^{(k)})$ with $v\neq y$, otherwise $c((y,H^{(k)}),a)=0$. Therefore, to every legitimate policy $f$ we can associate an expected total cost starting from state $(a_{0},H^{(k)}_{0})$, given by $h^{f}(y,y,H^{(k)})=0$, for any $H^{(k)}$ and, for any $a_{0}\neq y$ and any $H^{(k)}_{0}$,

[TABLE]

To be more clear, the expectations in equation (15) are over random variables $G_{1},G_{2},\ldots$, while the expectation in equation (16) is over $G_{2},G_{3},\ldots$. Furthermore, equation (16) follows by conditioning on $G_{1}$ and equation (17) follows by observing that, by symmetry, $\mathbb{E}^{f}\left[\sum_{t=1}^{\infty}c((a_{t},H^{(k)}_{t}),a_{t+1})\right]$ equals the expected total cost starting from $(a_{1},H^{(k)}_{1})$.

Observation 2**.**

Any* policy $f$ guiding Alice from $s$ to $y$ must satisfy recurrence (17), with initial condition $h^{f}(y,y,H^{(k)})=0$, for every $H^{(k)}$.*

Our objective is to find a policy that minimizes the expected total cost $h^{f}(a_{0},y,H^{(k)}_{0})$. In particular, this policy will have the value $h^{*}(a_{0},y,H^{(k)}_{0})=\inf_{f}h^{f}(a_{0},y,H^{(k)}_{0})$ which will be equal to the expected arrival time of a journey suggested to Alice by an optimal policy. In fact, without loss of generality we will assume that the $h^{*}$-values of an optimal policy satisfy $h^{*}(a_{0},y,H^{(k)}_{0})=\inf_{f}h^{f}(a_{0},y,H^{(k)}_{0})$, for all $a_{0}\in V$ and all $H^{(k)}_{0}=\left(G_{-k+1},G_{-k+2},\ldots,G_{-1},G_{0}\right)$, such that $G_{i}\subseteq G$, for all $-k+1\leq i\leq 0$.

4.3.2 A doubly exponential-time algorithm

We now provide our doubly exponential-time algorithm for Best Policy in the memory-$k$ model, where $k\geq 1$. In order to simplify the notation and presentation of this section, we only provide the proof of the algorithm for the special case $k=1$; the analysis for arbitrary $k\geq 1$ carries then easily over, as we discuss at the end of the section.

Following the notation of Section 4.3.1 for memory-1, we denote by $\mu(G^{\prime\prime}|G^{\prime})$ the probability that the next snapshot is $G^{\prime\prime}$, given that the current snapshot is $G^{\prime}$. Furthermore, for $a_{0}\in V$, let $h(a_{0},y,G_{0})$ be the expected arrival time of a journey from $a_{0}$ to $y$ suggested to Alice by an optimal policy, given that the starting graph instance is equal to $G_{0}$.

We define the following policy $\pi$: For any time step $t\geq 0$, if at time $t$ Alice was on a vertex $a_{t}$ and at time $t+1$ the graph instance is $G_{t+1}$, then at time $t+1$ she will move to a vertex $u\in\Gamma_{G_{t+1}}[a_{t}]$ that has minimum $h(u,y,G_{t+1})$, that is,

[TABLE]

By part (ii) of Theorem 5.4.3 of [32], we have the following:

Lemma 5**.**

Policy $\pi$ is optimal.

Notice that in the definition of $\pi$, we assumed that the $h$-values are given. Therefore, to determine $\pi$ we need to compute $h(a_{0},y,G_{0})$, for every $a_{0}\in V$ and $G_{0}\subseteq G$. We start by rewriting recurrence (17) for policy $\pi$:

[TABLE]

Since $\pi$ is optimal, the left hand side of the above equation is equal to $h(a_{0},y,G_{0})$. Furthermore, by definition of $\pi(a_{0},G_{1})$,

[TABLE]

Therefore, recurrence (19) becomes

[TABLE]

Suppose that we know an ordering of the triplets $(a_{0},y,G_{0}),a_{0}\in V,G_{0}\subseteq G$, in increasing values of $h(a_{0},y,G_{0})$, breaking ties arbitrarily. Notice that these are $n2^{m}\stackrel{{\scriptstyle\text{def}}}{{=}}N$ values, where $m=|E|$ is the number of edges of $G$. Then the minimum in recurrence (21) can be replaced with the corresponding $h$-value, which is completely determined by the graph $G_{1}$ and the vertex $a_{0}$. Doing this for all different vertices $a_{0}$ and graphs $G_{0}$, we get a linear system with $N$ equations coming from (21) and as many variables (the $h$-values). To this system, we then add the initial conditions $h(y,y,G_{0})=0$, for all $G_{0}\subseteq G$. This can be solved in $O(N^{3})$ time.

Notice however, that for the above approach to work, we need an ordering of the triplets $(a_{0},y,G_{0})$ in increasing values of $h(a_{0},y,G_{0})$. We can therefore have the following brute-force algorithm: For each of the (at most) $N!$ orderings of the triplets $(u_{0},y,G_{0}),a_{0}\in V,G_{0}\subseteq G$, solve the linear system derived by the recurrence (21) as described above, assuming the ordering is “correct”, namely it corresponds to an ordering in increasing values of $h(a_{0},y,G_{0})$. Then check if the ordering we get from the solution to that system is the same as the one we assumed. If not, then consider a different ordering.

Notice however the following issue with the above brute-force algorithm: suppose the correct ordering $\sigma^{*}$ is considered and we construct the corresponding linear system of equations (call it $\Sigma$) based on (21). Clearly, $\Sigma$ has at least one solution, but what happens if there are more than one solutions, some of which giving an ordering that is not consistent with $\sigma^{*}$? Can we find the correct solution among all other solutions of $\Sigma$? To circumvent this problem, we replace the linear system of equalities with a linear system of inequalities (constraints). To this end, let $u_{a_{0},G_{1}}^{\sigma^{*}}$ be the vertex such that $u_{a_{0},G_{1}}^{\sigma^{*}}\in\Gamma_{G_{1}}[a_{0}]$ and the triplet $(u_{a_{0},G_{1}}^{\sigma^{*}},y,G_{1})$ appears in $\sigma^{*}$ before all triplets $(u,y,G_{1})$, for all $u\in\Gamma_{G_{1}}[a_{0}]\setminus\{u_{a_{0},G_{1}}^{\sigma^{*}}\}$. We want to find any solution satisfying the following constraints:

[TABLE]

By definition of $\sigma^{*}$, the above set of constraints has at least one solution, namely the one corresponding to the $h$-values of an optimal policy. In Theorem 11 we prove that this is the only feasible solution. For the proof, we also need the following Theorem from [32], which we restate here in our notation for convenience:

Theorem 10** (Policy increment, Theorem 5.4.4, [32]).**

Given one stationary policy $f$, let $\theta f$ denote the policy that, for every $a_{0},G_{0}$ minimizes $\sum_{G_{1}}\mu(G_{1}|G_{0})h^{f}((\theta f)(a_{0},G_{1}),y,G_{1})$. Then, for all $a_{0},G_{0}$

[TABLE]

provided $\mathbb{E}_{(a_{0},G_{0})}h^{f}(a_{n},y,G_{n})\to 0$ as $n\to\infty$.

In the above, the notation $\theta^{k}f$ means the application of policy increment $k$ times. Furthermore, in the expectation $\mathbb{E}_{(a_{0},G_{0})}h^{f}(a_{n},y,G_{n})$, the state $(a_{n},G_{n})$ is a random variable and its distribution is determined by an optimal policy, given that we start at $(a_{0},G_{0})$. We note that, the condition of the Theorem holds in our case by transience of the underlying Markov chain (i.e. once Alice reaches $y$ she does not leave and no further cost is incurred after that).

We now prove the following:

Theorem 11**.**

Let $\sigma^{*}$ be an ordering of the triplets $(a_{0},y,G_{0}),a_{0}\in V,G_{0}\subseteq G$, in increasing order of $h$-values. Let $(h^{\prime*}(a_{0},y,G_{0}):a_{0}\in V,G_{0}\subseteq G)$ be any feasible solution to the set of constraints (22) to (25). Then $h^{\prime*}(a_{0},y,G_{0})=h(a_{0},y,G_{0})$, for all $a_{0},G_{0}$.

Proof.

We define the following policy $\pi^{*}$ (similar to the definition of $\pi$ earlier, but using the $h^{\prime*}$-values instead of the $h$-values): For any time step $t\geq 0$, if at time $t$ Alice was on a vertex $a_{t}$ and at time $t+1$ the graph instance is $G_{t+1}$, then at time $t+1$ she will move to a vertex $u\in\Gamma_{G_{t+1}}[a_{t}]$ that has minimum $h^{\prime*}(u,y,G_{t+1})$, that is,

[TABLE]

Since the $h^{\prime*}$-values satisfy constraint (22) and also, by constraint (23), $u_{a_{0},G_{1}}^{\sigma^{*}}\in\arg\min\{h^{\prime*}(u,y,G_{1}):u\in\Gamma_{G_{1}}[a_{0}]\}$, the expected arrival time of a journey from $a_{0}$ to $y$ when Alice follows policy $\pi^{*}$ is equal to $h^{\prime*}(a_{0},y,G_{0})$, for all $a_{0}\in V$ and $G_{0}\subseteq G$.

Observe that, if $\pi^{*}$ is not optimal, then we can successively apply policy increments $\theta^{k}\pi^{*},k\to\infty$ as in Theorem 10, to eventually reach optimality. Notice also that the sum in the definition of policy increment for $\pi^{*}$ can be written as

[TABLE]

Therefore, by definition, $\pi^{*}$ itself is a policy that minimizes the above sum, and so we can take $\theta\pi^{*}=\pi^{*}$. Consequently, no improvement by increment is possible, implying that $\pi^{*}$ is optimal. In particular, $(h^{\prime*}(a_{0},y,G_{0}):a_{0}\in V,G_{0}\subseteq G)$ is the same as $(h(a_{0},y,G_{0}):a_{0}\in V,G_{0}\subseteq G)$, and the proof is completed. ∎

The set of constraints (22) to (25) has $N=n2^{m}$ variables, namely $\{h^{\prime}(a_{0},y,G_{0}):a_{0}\in V,G_{0}\subseteq G\}$. Furthermore, there are $(n-1)2^{m}$ constraints of the form (22), at most $n^{2}2^{m}$ constraints of the form (23) and $n2^{m}$ non-negativity and initialization constraints, i.e. $O(nN)$ constraints in total. Therefore, Vaydia’s algorithm for linear programming [38] can find an optimum solution in $O((nN)^{2.5})$ time. Since we need to solve this set of constraints for every possible ordering of the $N$ different triplets $(a_{0},y,G_{0})$, our brute-force approach runs in $O(N!\ (nN)^{2.5})=O(N^{N})$ time.

The above analysis for the memory-1 model directly carries over to the memory-$k$ model, for any $k\geq 1$. Indeed, the correctness proof can be slightly modified by replacing everywhere the subgraphs $G_{t}$ of $G$ by the length-$k$ histories $H^{(k)}_{t}$, respectively. Furthermore, the running time analysis carries over to the case of an arbitrary $k\geq 1$ by replacing $N=n2^{m}$ by $N^{\prime}=n2^{km}$. Summarizing, we obtain the following theorem.

Theorem 12**.**

Let $k\geq 1$ and $\mathcal{G}^{(k)}$ be a stochastic temporal graph, where the underlying graph $G$ has $n$ vertices and $m$ edges. Then Best Policy can be solved on $\mathcal{G}^{(k)}$ in $O(2^{(kmn+n\log n)\cdot 2^{km}})$ time.

Remark 1**.**

It is easy to see that the running time of the above brute-force algorithm is dominated by the number of different orderings $N^{\prime}!$, and thus we have a doubly exponential algorithm (recall that $N^{\prime}=n2^{km}$). A different approach that can potentially lead to a faster algorithm is to start from an arbitrary initial policy and successively apply policy increments as in Theorem 10. Even though the convergence analysis of such an approach is non-trivial, one could use it to find the optimal ordering $\sigma^{*}$ fast555This is possible if after a relatively small (say polynomial in $N^{\prime}$) number of steps the ordering does not change. and then use $\sigma^{*}$ to find the unique solution to the set of constraints (22)-(25).