Timely Updates in Energy Harvesting Two-hop Networks: Offline and Online Policies

TL;DR

This paper studies optimal update policies in energy harvesting two-hop networks to minimize information age, proposing solutions for both known and unknown energy arrival scenarios, emphasizing timely, uniform update transmission.

Contribution

It introduces optimal offline and online policies for energy harvesting two-hop networks, demonstrating near-uniform update scheduling under energy and delay constraints.

Findings

Optimal policies transmit updates just in time for relay readiness.

Offline solutions achieve perfectly uniform update intervals.

Online policies approximate uniformity using a best effort approach.

Abstract

A two-hop energy harvesting communication network is considered, in which measurement updates are transmitted by a source to a destination through an intermediate relay. Updates are to be sent in a timely fashion that minimizes the age of information, defined as the time elapsed since the most recent update at the destination was generated at the source. The source and the relay communicate using energy harvested from nature, which is stored in infinite-sized batteries. Both nodes use fixed transmission rates, and hence updates incur fixed delays (service times). Two problems are formulated: an offline problem, in which the energy arrival information is known a priori, and an online problem, in which such information is revealed causally over time. In both problems, it is shown that it is optimal to transmit updates from the source just in time as the relay is ready to forward them to the destination, making the source and the relay act as one combined node. A recurring theme in the optimal policy is that updates should be as uniformly spread out over time as possible, subject to energy causality and service time constraints. This is perfectly achieved in the offline setting, and is achieved almost surely in the online setting by a best effort policy.

Equations

$$a(t)\triangleq t-u(t)$$

$$A_T\triangleq {\int }_0^{T}a(t)dt.$$

$$t_i\ge s_i,{\bar{t}}_i\ge {\bar{s}}_i,1\le i\le N,$$

$$t_i+d\le {\bar{t}}_i,1\le i\le N,$$

$$t_i+d\le t_{i+1},{\bar{t}}_i+\bar{d}\le {\bar{t}}_{i+1},1\le i\le N-1,$$

$$(t_i,t_i+d)\cap ({\bar{t}}_j,{\bar{t}}_j+\bar{d})=\varnothing ,\forall i,j,$$

$${\bar{t}}_i+\bar{d}\le t_{i+1},1\le i\le N-1.$$

$${\bar{t}}_N+\bar{d}\le T.$$

$$\underset{\mathbf{t},\bar{\mathbf{t}}}{\min }$$

$$t_i\ge s_i,{\bar{t}}_i\ge {\bar{s}}_i,1\le i\le N$$

$$t_i+d\le {\bar{t}}_i,1\le i\le N$$

$${\bar{t}}_i+\bar{d}\le t_{i+1},1\le i\le N.$$

$$T$$

$$\underset{T\to \infty }{\mathrm{limsup}}\frac{1}{T}\mathbb{E}\left[A_T\right].$$

$$\mathcal{E}(t_i)$$

$$\bar{\mathcal{E}}\left({\bar{t}}_i\right)$$

$$\mathcal{E}(t_i)\ge E,\bar{\mathcal{E}}\left({\bar{t}}_i\right)\ge \bar{E},\forall i.$$

$$\rho \triangleq \underset{\mathbf{t},\bar{\mathbf{t}}\in \mathcal{F}}{\min }\underset{T\to \infty }{\mathrm{limsup}}\frac{1}{T}\mathbb{E}\left[A_T\right].$$

$$\underset{\mathbf{t}}{\min }$$

$$t_i\ge s_i,1\le i\le N$$

$$t_i+d\le t_{i+1},1\le i\le N,$$

$$\underset{\mathbf{x}}{\min }$$

$$\sum _{i=1}^k{x}_i\ge s_k+kd,1\le k\le N$$

$$x_i\ge 2d,2\le i\le N$$

$$x_{N+1}\ge d$$

$$\sum _{i=1}^{N+1}{x}_i=T+Nd.$$

$$\mathcal{L}=$$

$$+\nu \left(\sum _{i=1}^{N+1}{x}_i-T-Nd\right),$$

$$x_1$$

$$x_i$$

$$x_{N+1}$$

$${\lambda }_k\left(\sum _{i=1}^k{x}_i-s_k-kd\right)$$

$${\eta }_i\left(x_i-2d\right)$$

$${\eta }_{N+1}(x_{N+1}-d)$$

$$x_1^{\ast }$$

$$x_i^{\ast }$$

$$x_{N+1}^{\ast }$$

$$\underset{x_1}{\min }$$

Full text

I Introduction

Developing low-latency communication policies is a crucial requirement for next generation communication networks, especially in applications pertaining to real-time status monitoring where information needs to be kept as fresh as possible at interested destinations. The age of information (AoI) metric has been recently introduced in the literature as a suitable performance metric to assess the freshness of data through basically measuring delays from receivers’ perspectives. In applications where measurement updates regarding some phenomenon are to be sent to some destination over a period of time, the AoI is defined as the time elapsed since the most recent update at the destination was generated at the source. While it is clearly optimal to keep sending updates to maintain a low AoI, this might not be feasible all the time if transmitters have limited energy budgets. Therefore, managing the available energy becomes critical. In this paper, we consider an energy harvesting two-hop network, and propose optimal update policies that minimize the AoI in offline settings, where the energy arrival information is known prior to the start of communication, and online settings, where such information is only revealed causally over time.

Minimizing AoI has been extensively considered in the literature under various system settings and assumptions. Interest in AoI started mainly in queuing systems, in which the early work of [1] shows that minimizing AoI is neither equivalent to minimizing transmission delay nor to maximizing throughput in a single server first-come first-serve queue with an infinite data buffer. In particular, it is optimal to design packet arrival rates in such systems to have an intermediate value between low arrival rates (that minimize transmission delay) and high arrival rates (that maximize throughput). One main reason behind this is that to minimize AoI one needs to keep the queue size relatively small, and at the same time keep delivering packets at a reasonable pace. This fundamental observation was then extended to the case of multiple sources in [2], and more recently in [3]. References [4, 5, 6] consider variations of the single source system and the queuing disciplines, such as randomly (out of order) arriving updates, update management and control, and nonlinear AoI metrics. Other variations considered include broadcasting, multicasting and multi streaming [7, 8, 9]. Reference [10] derives conditions on when it is optimal to submit new status updates right away after previous ones are delivered, and shows that, interestingly, it is not always optimal to do so; depending on the service time distribution it might be better to wait for a period of time before submitting the new update. Timely coding over erasure channels is studied in [11]; connections between AoI and caching systems are drawn in [12]; and those between AoI and remote estimation are drawn in [13]. Different from the single-hop settings considered in most works, references [14, 15, 16, 17, 18] focus on multihop settings. For instance, it is shown in [16] that last-generated first-served policies are optimal in multihop networks under fairly arbitrary network topologies and packet arrival distributions. We refer the reader to the detailed survey in [19] for more details on the various contexts in which AoI has been used.

Another setting in which AoI minimization has recently been studied is when transmitters/sensors rely on energy harvested from nature to communicate [20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36]. Energy harvesting offer the promise of providing energy self-sufficient and self-sustaining means of communications, and has been thoroughly studied in various contexts in the recent literature, see, e.g., [37, 38]. In such settings, updates cannot be sent all the time, and optimal (AoI-minimal) energy management policies need to be carefully designed. The differentiating aspects between the works in [20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36] are mainly: battery capacity, offline/online knowledge of the energy arrivals, and service times (times for the updates to take effect). Aside from the conference version of this current work [22], which mainly discussed the offline setting, the above works focus on single-hop settings under both perfect and imperfect (noisy) channel conditions. A general conclusion from these works as a whole is that to minimize AoI, one needs to send status updates that are uniformly spread-out over time, to the extent allowed by the energy harvesting constraints. It is worth mentioning that the difference between AoI and delay minimization in energy harvesting communications has been discussed in [23, Section V].

In this paper, we characterize age-minimal status update policies in energy harvesting two-hop networks, in which a source is communicating with a destination through the help of an intermediate relay node. We consider the setting in which both the source and the relay are equipped with infinite batteries to save their incoming energy; offline and online knowledge of the energy arrivals; and fixed service times. In the offline scenario, the goal is to minimize the average AoI by a given deadline. We first solve a single-hop version of the problem, and then use its solution to solve for the two-hop version. Specifically, we show that it is optimal for the source to transmit a new update just in time as the relay is ready to forward it to the destination. Thus, the source and the relay can be treated as one combined node, after some appropriate transformations. The solution of the offline problem follows via an inter-update balancing algorithm, which aims at uniformly spreading the updates over time, up to the extent allowed by the energy arrivals and service times. In the online scenario, the goal is to minimize the long term average AoI. Energy arrives at the source and the relay according to two independent Poisson processes. We also show in this case that it is age-minimal to treat the source and the relay as one combined node. We then propose a best effort uniform update policy with service constraints, in which time is divided into slots of equal durations that depend on the service times, and an update is transmitted at the beginning of each time slot only if both the source and the relay have enough energy, otherwise both nodes stay silent and re-attempt transmission at the beginning of the next time slot. We prove that this policy is age-minimal by showing that it achieves a lower bound on the long term average AoI almost surely.

We note that the optimality of treating the source and the relay nodes as one combined node, and its consequences on not keeping any update packets waiting in the relay’s data buffer, echoes the optimality results of last-generated first-served policies in the multihop setting considered in [16]. We also note that such result is in contrast to the optimality of separable policies that maximize throughput in energy harvesting two-hop networks [39, 40], where it is optimal to treat each node independently and send as many packets as possibly allowed by its own energy arrivals without considering those of the other node. In that sense, this work shows that if the metric considered is AoI, optimal policies are inseparable.

II System Model and Problem Formulation

A source node acquires measurement updates from some physical phenomenon and sends them to a destination, through the help of a half-duplex relay, see Fig. 1. Both the source and the relay depend on energy harvested from nature to transmit their data, and are equipped with infinite-sized batteries to save their incoming energy. Energy arrives in packets of amounts $E$ and $\bar{E}$ at the source and the relay, respectively. Update packets are of equal length, and are transmitted using fixed rates at the source and the relay. We assume that one update transmission consumes one energy packet at a given node, and hence the number of updates at a given time is equal to the minimum of the number of energy packets that arrived at the source and the relay by that time. Under a fixed rate policy, each update takes $d$ and $\bar{d}$ amount of time to get through the source-relay channel and the relay-destination channel, respectively.111We focus on error-free channels in this first look at multihop AoI minimization problems under energy harvesting constraints, in order to provide a clear understanding of the optimal status update policies’ structure. We note that other works, e.g., [26, 27, 35, 36] have considered noisy channels situations for single-hop settings, in which updates are subject to erasures. Different from references [26, 27, 35, 36] though, which focus on zero (negligible) service times, we focus on non-zero, fixed, service times in the multihop setting of this work.

The goal is to send updates as timely as possible, namely, such that the age of information (AoI) is minimized. The AoI metric at time $t$ is defined as

[TABLE]

where $u(t)$ is the time stamp of the latest received update packet at the destination before time $t$, i.e., the time at which it was acquired at the source.

We note that the fixed energy consumption considerations in this work are derived by the fact that each status update typically consists of a value of the measurement acquired, and a time stamp denoting its acquisition time, that are both encoded into fixed-size packets of length, e.g., $B$ bits. The energy consumed to sense, encode, and transmit a single status update packet at the source is assumed to be equal to $E$, while the energy consumed to decode and forward a single status update packet at the relay is assumed to be equal to $\bar{E}$. Operating at such fixed energy consumption levels induces fixed transmission rate policies. For instance, if $g(e)$ is the rate in, e.g., bits per time units, used to send one status update packet with energy consumption $e$, then such update will take $B/g(e)$ time units of transmission delay to traverse through the communication channel. In our setting, such transmission delay is equal to $d$ and $\bar{d}$ at the source-relay channel and the relay-destination channel, respectively. The case in which transmitting devices are capable of using variable rates for different status updates has been previously considered in the single-hop setting in [23]. In this current work, however, we focus on fixed rates (and therefore fixed transmission delays) in order to fully characterize the optimal AoI minimizing solutions for multihop settings under both offline and online future energy arrival knowledge, which are formally introduced in the next subsections.

II-A The Offline Setting

In the offline setting, we consider a communication session of duration $T$ time units, in which energy arrival times are known a priori. Without loss of generality, we assume $a(0)=0$. The objective is to minimize the following quantity:

[TABLE]

Source energy packets arrive at times $\{s_{1},s_{2},\dots,s_{N}\}\triangleq{\bm{s}}$, and relay energy packets arrive at times $\{\bar{s}_{1},\bar{s}_{2},\dots,\bar{s}_{N}\}\triangleq\bar{\bm{s}}$, where without loss of generality we assume that both the source and the relay receive $N$ energy packets, since each update consumes one energy packet in transmission from either node, and hence any extra energy arrivals at either the source or the relay cannot be used. Let $t_{i}$ and $\bar{t}_{i}$ denote the transmission time of the $i$th update at the source and the relay, respectively. We first impose the following constraints:

[TABLE]

representing the energy causality constraints [41] at the source and the relay, which mean that no energy packet can be used before being harvested. Next, we must have

[TABLE]

to ensure that the relay does not forward an update before receiving it from the source, which represents the data causality constraints [41]. We also have the service time constraints

[TABLE]

which ensure that there can only be one transmission at a time at the source and the relay. Hence, $d$ and $\bar{d}$ represent the service (busy) time of the source and relay servers, respectively.

Transmission times at the source and the relay should also be related according to the half-duplex nature of the relay operation. For that, we must have the half-duplex constraints

[TABLE]

where $\emptyset$ denotes the empty set, since the relay cannot receive and transmit simultaneously. These constraints enforce that either the source transmits a new update after the relay finishes forwarding the prior one, i.e., $t_{i+1}\geq\bar{t}_{i}+\bar{d}$ for some $i$; or that the source delivers a new update before the relay starts transmitting the prior one, i.e., $t_{i+k}+d\leq\bar{t}_{i}$ for some $i$ and $k$. The latter case means that there are $k+1$ update packets waiting in the relay’s data buffer just before time $\bar{t}_{i}$. We prove that this case is not age-optimal. To see this, consider the example of having $k+1=2$ update packets in the relay’s data buffer waiting for service. The relay in this case has two choices at its upcoming transmission time: 1) forward the first update followed by the second one sometime later, or 2) forward the second update only and ignore the first one. These two choices yield different age evolution curves. We observe, geometrically, that $A_{T}$ under choice 2 is strictly less than that under choice 1. Since the source under choice 2 consumes an extra energy packet to send the first update unnecessarily, it should instead save this energy packet to send a new update after the first one is forwarded by the relay. Therefore, it is optimal to replace the half-duplex constraints in (6) by the following reduced ones:

[TABLE]

Next, observe that (5) can be removed from the constraints since it is implied by (4) and (7). In conclusion, the constraints are now those in (3), (4), and (7). Finally, we add the following constraint to ensure reception of all updates by time $T$:

[TABLE]

In Fig. 2, we present an example of the age of information in a system with 3 updates. The area under the curve representing $A_{T}$ is given by the sum of the areas of the trapezoids $Q_{1}$, $Q_{2}$, and $Q_{3}$, in addition to the area of the triangle $L$. The area of $Q_{2}$ for instance is given by $\frac{1}{2}\left(\bar{t}_{2}+\bar{d}-t_{1}\right)^{2}-\frac{1}{2}\left(\bar{t}_{2}+\bar{d}-t_{2}\right)^{2}$. The objective is to choose feasible transmission times for the source and the relay such that $A_{T}$ is minimized. Computing the area under the age curve for general $N$ arrivals, we formulate the offline problem as follows:

[TABLE]

with $t_{0}\triangleq 0$ and $t_{N+1}\triangleq T$.

We note that the energy arrival times ${\bm{s}}$ and $\bar{\bm{s}}$, the transmission delays $d$ and $\bar{d}$, the session time $T$, and the number of energy arrivals $N$, are such that problem (II-A) has a feasible solution. This is true only if

[TABLE]

where (10) (resp. (11)) ensures that the $i$th energy arrival time at the relay (resp. source) is small enough to allow the reception of the upcoming $N-i$ updates within time $T$.

II-B The Online Setting

In the online setting, future energy arrival times are not known a priori; they get revealed causally over time. We assume, however, that there is some central controller that observes both energy arrival processes at the source and the relay, and takes decisions based on these simultaneous observations222A completely decentralized system model in which the source takes scheduling decisions independently from the relay is a possible direction for future work. In this work, we focus on the centralized problem.. The energy arrival processes at the source and the relay are modeled as two independent Poisson processes of unit rate. The objective is to minimize the long term average area under the AoI curve. That is, to minimize

[TABLE]

Let us denote by $\mathcal{E}(t)$ and $\bar{\mathcal{E}}(t)$ the energy available in the source’s and relay’s batteries, respectively, right before time $t$, i.e., at time $t^{-}$. Therefore, the batteries evolve as follows:

[TABLE]

where $\mathcal{A}\left([a,b)\right)$ and $\bar{\mathcal{A}}\left([a,b)\right)$ denote the total number of energy arrivals in the interval $[a,b)$ at the source and the relay, respectively, which are two independent Poisson random variables with parameter $b-a$. We now write the energy causality constraints slightly differently from the offline setting as follows:

[TABLE]

Let the feasible set $\mathcal{F}$ be the set of transmission times $\{t_{i}\}$ and $\{\bar{t}_{i}\}$ that adhere to the constraints in (4), (7) and (13)-(15), in addition to the following at time [math]: $\mathcal{E}(0)=E$ and $\bar{\mathcal{E}}(0)=\bar{E}$. The online problem is now to characterize the following quantity:

[TABLE]

We discuss problems (II-A) and (16) in details over the next two sections.

III The Offline Problem

III-A Solution Building Block: The Single-User Channel

In this subsection, we solve the single-user version of problem (II-A); namely, when the source is communicating directly with the destination. We use the solution to the single-user problem in this subsection as a building block to solve problem (II-A) in the next subsection. In Fig. 3, we show an example of the age evolution in a single-user setting. The area of $Q_{2}$ is now given by $\frac{1}{2}\left(t_{2}+d-t_{1}\right)^{2}-\frac{1}{2}d^{2}$. We compute the area under the age curve for general $N$ arrivals and formulate the single-user problem as follows:

[TABLE]

where the second constraints are the service time constraints.

We note that reference [21] considered problem (III-A) when the transmission delay $d=0$. We extend their results for a positive delay (and hence a finite transmission rate) in this subsection. We first introduce the following change of variables: $x_{1}\triangleq t_{1}+d$; $x_{i}\triangleq t_{i}-t_{i-1}+d,~{}2\leq i\leq N$; and $x_{N+1}\triangleq T-t_{N}$. These variables must satisfy $\sum_{i=1}^{N+1}x_{i}=T+Nd$, which reflects the dependent relationship between the new variables $\{x_{i}\}$. This can also be seen from Fig. 3. Substituting by $\{x_{i}\}$ in problem (III-A), we get the following equivalent problem:

[TABLE]

Observe that problem (III-A) is a convex problem that can be solved by standard techniques [42]. For instance, we introduce the following Lagrangian:

[TABLE]

where $\{\lambda_{1},\dots,\lambda_{N},\eta_{2}\dots,\eta_{N+1},\nu\}$ are Lagrange multipliers, with $\lambda_{i},\eta_{i}\geq 0$ and $\nu\in\mathbb{R}$. Differentiating with respect to $x_{i}$ and equating to 0 we get the following KKT conditions:

[TABLE]

along with complementary slackness conditions

[TABLE]

We now have the following lemmas characterizing $\{x_{i}^{*}\}$, the optimal solution of problem (III-A). Lemmas 1 and 3 show that the sequence $\{x_{i}^{*}\}_{i=2}^{N+1}$ is non-increasing, and derive necessary conditions for it to strictly decrease. On the other hand, Lemma 2 shows that $x_{1}^{*}$ can be smaller or larger than $x_{2}^{*}$, and derives necessary conditions for the two cases.

Lemma 1

For $2\leq i\leq N-1$, $x_{i}^{*}\geq x_{i+1}^{*}$. Furthermore, $x_{i}^{*}>x_{i+1}^{*}$ only if $\sum_{j=1}^{i}x_{j}^{*}=s_{i}+id$.

**Proof: ** We show this by contradiction. Assume that for some $i\in\{2,\dots,N-1\}$ we have $x_{i}^{*}<x_{i+1}^{*}$. By (21), this is equivalent to having $\lambda_{i}+\eta_{i}<\eta_{i+1}$, i.e., $\eta_{i+1}>0$, which implies by complementary slackness in (24) that $x_{i+1}^{*}=2d$. This means that $x_{i}^{*}<2d$, i.e., infeasible. Therefore $x_{i}^{*}\geq x_{i+1}^{*}$ holds. This proves the first part of the lemma.

To show the second part, observe that since $x_{i}^{*}>x_{i+1}^{*}$ holds if and only if $\lambda_{i}+\eta_{i}>\eta_{i+1}$, then either $\lambda_{i}>0$ or $\eta_{i}>0$. If $\eta_{i}>0$, then by (24) we must have $x_{i}^{*}=2d$, which renders $x_{i+1}^{*}<2d$, i.e., infeasible. Therefore, $\eta_{i}$ cannot be positive and we must have $\lambda_{i}>0$. By complementary slackness in (23), this implies that $\sum_{j=1}^{i}x_{j}^{*}=s_{i}+id$.  $\blacksquare$

Lemma 2

$x_{1}^{*}>x_{2}^{*}$* only if $x_{1}^{*}=s_{1}+d$; while $x_{1}^{*}<x_{2}^{*}$ only if $x_{i}^{*}=2d$, for $2\leq i\leq N$.*

**Proof: ** The necessary condition for $x_{1}^{*}$ to be larger than $x_{2}^{*}$ can be shown using the same arguments as in the proof of the second part of Lemma 1, and is omitted for brevity. Let us now assume that $x_{1}^{*}$ is smaller than $x_{2}^{*}$. By (20) and (21), this occurs if and only if $\eta_{2}>\lambda_{1}$, which implies that $x_{2}^{*}=2d$ by complementary slackness in (24). Finally, by Lemma 1, we know that $\{x_{i}^{*}\}_{i=2}^{N}$ is non-increasing; since they are all bounded below by $2d$, and $x_{2}^{*}=2d$, then they must all be equal to $2d$.  $\blacksquare$

Lemma 3

$x_{N}^{*}\geq x_{N+1}^{*}$. Furthermore, $x_{N}^{*}>x_{N+1}^{*}$ only if at least: 1) $\sum_{i=1}^{N}x_{i}^{*}=s_{N}+Nd$, or 2) $x_{N}^{*}=2d$ occurs.

The proof of Lemma 3 is along the same lines of the proofs of the previous two lemmas and is omitted for brevity.

Observe that Lemma 1 states that (intermediate) inter-update delays cannot be increasing, and can only decreases following an update that was transmitted right away after an energy unit is harvested. Lemma 2 states that an exception holds for the second inter-update delay, which can only increase, with respect to the first one, only if all updates starting from the second one to the end are sent back-to-back. Finally, Lemma 3 adds another condition to that of Lemma 1 for the last inter-update delay to decrease, which is to send the last update back-to-back after the previous one.

We will use the results of Lemmas 1, 2, and 3 to derive the optimal solution of problem (III-A). To do so, one has to consider the relationship between the parameters of the problem: $T$, $d$, and $N$. For instance, one expects that if the session time $T$ is much larger than the minimum inter-update time $d$, then the energy causality constraints will be binding while the constraints enforcing one update at a time will not be, and vice versa. We formalize this idea by considering two different cases as follows.

III-A1 $Nd\leq T<(N+1)d$

We first note that $Nd$ is the least value that $T$ can have for problem (III-A) to admit a feasible solution. In this case, the following theorem shows that the optimal solution is achieved by sending all updates back-to-back with the minimal inter-update time possible to allow the reception of all of them by the end of the relatively small session time $T$.

Theorem 1

Let $Nd\leq T<(N+1)d$. Then, the optimal solution of problem (III-A) is given by

[TABLE]

**Proof: ** We first argue that if $x_{1}^{*}\geq x_{2}^{*}(\geq 2d)$, then $\sum_{i=1}^{N+1}x_{i}^{*}\geq(2N+1)d$. The last constraint in problem (III-A) then implies that $T\geq(N+1)d$, which is infeasible in this case. Therefore, we must have $x_{1}^{*}<x_{2}^{*}$. By Lemma 2, this occurs only if $x_{i}^{*}=2d$ for $2\leq i\leq N$. Hence, we set $x_{N+1}=T-(N-2)d-x_{1}$, and observe that problem (III-A) in this case reduces to a problem in only one variable $x_{1}$ as follows:

[TABLE]

whose solution is given by projecting the critical point of the objective function onto the feasible interval since the problem is convex [42]. This directly gives (26).  $\blacksquare$

III-A2 $T\geq(N+1)d$

In this case, we propose an algorithmic solution that is based on the necessary optimality conditions in Lemmas 1, 2, and 3. We first solve problem (III-A) without considering the service time constraints, i.e., assuming that the set of constraints $\{x_{i}\geq 2d,~{}2\leq i\leq N;~{}x_{N+1}\geq d\}$ is not active. We then check if any of these abandoned constraints is not satisfied, and optimally alter the solution to make it feasible.

Let us denote by $\left(P^{e}\right)$ problem (III-A) without the set of constraints $\{x_{i}\geq 2d,~{}2\leq i\leq N;~{}x_{N+1}\geq d\}$, i.e., considering only the energy causality constraints. We then introduce the following algorithm to solve problem $\left(P^{e}\right)$:

Definition 1** **(Inter-Update Balancing Algorithm)

Start by computing

[TABLE]

where the set is indexed as $\{1,\dots,N+1\}$, and then set

[TABLE]

If $i_{1}=N+1$ stop, else compute

[TABLE]

where the set is indexed as $\{i_{1}+1,\dots,N+1\}$, and then set

[TABLE]

If $i_{2}=N+1$ stop, else continue with computing $i_{3}$ as above. The algorithm is guaranteed to stop since it will at most compute $i_{N+1}$ which is equal to $N+1$ by construction.

Note that while computing $i_{k}$, if the $\arg\max$ is not unique, we pick the largest maximizer. Observe that the algorithm equalizes the $x_{i}$’s as much as allowed by the energy causality constraints. In Fig. 4, we illustrate how the algorithm works using a graphical example. In the example, we have $N=3$ energy arrivals at $s_{1}=3$, $s_{2}=10$, and $s_{3}=12$ time units. Service time $d=4$ time units, and the session ends at $T=20$ time units. Applying the first run of the algorithm gives $i_{1}=2$. Therefore, we have $\bar{x}_{1}=\bar{x}_{2}=\frac{s_{2}}{2}+d=9$. Since $i_{1}=2<N+1=4$, the algorithm continues for a second run. Note that after the first run, the system has only one arrival left (at $s_{3}$), and it is as if the algorithm restarts again with a new reference point at $s_{2}$, as opposed to [math]. Applying the algorithm starting from $s_{2}$ in the second run gives $i_{2}=4$, which means $\bar{x}_{3}=\bar{x}_{4}=\frac{T-d-s_{2}}{N+1}+d=8$. Since $i_{2}=N+1=4$, the algorithm terminates.

Let $\{\bar{x}_{i}\}_{i=1}^{N}$ be the output of the Inter-Update Balancing algorithm and let $\{x_{i}^{e}\}_{i=1}^{N}$ denote the optimal solution of problem $(P^{e})$. We now have the following results:

Lemma 4

$\{\bar{x}_{i}\}_{i=1}^{N}$* is a non-increasing sequence, and $\bar{x}_{j}>\bar{x}_{j+1}$ only if $\sum_{i=1}^{j}\bar{x}_{i}=s_{j}+jd$.*

Lemma 5

$x_{i}^{e}=\bar{x}_{i}$, $1\leq i\leq N$.

The proofs of Lemmas 4 and 5 are in Appendices -A and -B, respectively.

We note that Lemma 4 can be interpreted, operationally and in effect, as Lemma 1. We also note that Lemma 5 is similar to [21, Theorem 1]. In fact, the Inter-Update Balancing algorithm reduces to the optimal offline algorithm proposed in [21] when $d=0$. When $d>0$, some change of parameters can still show the equivalence. The next corollary now follows.

Corollary 1

Consider problem $(P^{e})$ with the additional constraint that $\sum_{i=1}^{j}x_{i}=s_{j}+jd$ holds for some $j\leq N$. Then, the optimal solution of the problem, under this condition, for time indices not larger than $j$ is given by $\{x_{i}^{e}\}_{i=1}^{j}$.

**Proof: ** This is direct by setting $T^{\prime}\triangleq s_{j}+d$ and $N^{\prime}\triangleq j-1$, and applying the Inter-Update Balancing algorithm on the problem with a reduced number of variables $\{x_{1},\dots,x_{N^{\prime}+1}\}$.  $\blacksquare$

The following theorem shows that the optimal solution of problem (III-A), $\{x_{i}^{*}\}$, is found by equalizing the inter-update times as much as allowed by the energy causality constraints. If such equalization does not satisfy the minimal inter-update time constraints, we force it to be exactly equal to such minimum and adjust the last variable $x_{N+1}$ accordingly. The proof of the theorem is in Appendix -C.

Theorem 2

Let $T\geq(N+1)d$. If $x_{i}^{e}\geq 2d,~{}2\leq i\leq N$ and $x_{N+1}^{e}\geq d$, then $x_{i}^{*}=x_{i}^{e},~{}\forall i$. Else, let $n_{0}$ be the first time index at which $\{x_{i}^{e}\}$ is not feasible in problem (III-A). Then, we have $n_{0}\leq N$. If $n_{0}>2$, we have

[TABLE]

Otherwise, for $n_{0}=2$, $\{x_{i}^{*}\}$ is given by the above if $x_{1}^{e}=s_{1}+d$, else $\{x_{i}^{*}\}$ is given by (26)-(28).

III-B Two-Hop Network: Solution of Problem (II-A)

We now discuss how to use the results of the single-user problem to solve problem (II-A). We have the following theorem:

Theorem 3

*The optimal solution of problem (II-A) is given by the optimal solution of problem (III-A) after replacing $s_{i}$ by $\max\{\bar{s}_{i},s_{i}+d\},~{}\forall i$; $d$ by $d+\bar{d}$; and $T$ by $T+d$. *

**Proof: ** Let $f$ denote the objective function of problem (II-A). Differentiating $f$ with respect to $t_{i}$, $i\leq N-1$, we get $\frac{\partial f}{\partial t_{i}}=2\left(\bar{t}_{i}+\bar{d}-t_{i}\right)-2\left(\bar{t}_{i+1}+\bar{d}-t_{i}\right)$, which is negative since $\bar{t}_{i+1}>\bar{t}_{i}$. We also have $\frac{\partial f}{\partial t_{N}}=2\left(\bar{t}_{N}+\bar{d}-t_{N}\right)-2\left(T-t_{N}\right)$, which is non-positive since $\bar{t}_{N}+\bar{d}\leq T$. Thus, $f$ is decreasing in $\{t_{i}\}_{i=1}^{N-1}$ and non-increasing in $t_{N}$. Therefore, the optimal $\{t_{i}^{*}\}$ satisfies the data causality constraints in (4) with equality for all updates so as to be the largest possible and achieve the smallest $A_{T}$. Setting $t_{i}=\bar{t}_{i}-d,~{}\forall i$ in problem (II-A) we get

[TABLE]

with the constraints now being

[TABLE]

We now see that minimizing $f$ subject to the above constraints is exactly the same as solving problem (III-A) after applying the change of parameters mentioned in the theorem.  $\blacksquare$

Theorem 3 shows that the source should send its updates just in time as the relay is ready to forward, and no update should wait for service in the relay’s data buffer. Thus, the source and the relay act as one combined node that can send updates whenever it receives combined energy packets at times $\left\{\max\{\bar{s}_{i},s_{i}+d\}\right\}$. We show how to apply the theorem’s premises on a graphical example in Fig. 5. This fundamental observation can be generalized to multi-hop networks as well. Given $M>1$ relays, each node should send updates just in time as the following node is ready to forward, until reaching destination.

IV The Online Problem

In this section, we focus on characterizing the optimal solution of problem (16). We follow a two-step approach: we first derive a lower bound on the objective function of the problem, and then propose an online-feasible policy that achieves that lower bound, which shows its optimality. Let the random variable $N(T)$ denote the number of updates received by the destination within time $T$. We first begin by stating an upper bound on the update rate $N(T)/T$ in the limit as $T$ grows infinitely large in the following lemma:

Lemma 6

The following upper bound holds:

[TABLE]

**Proof: ** By ignoring the energy causality constraints, and since an update can be received at the destination if and only if both the source and the relay have energy, we have

[TABLE]

Since both energy arrival rates at the source and the relay are unit-valued, dividing both sides of the above equation by $T$ and taking limits shows that the update rate is upper-bounded by $1$. On the other hand, the service rate constraints imply that

[TABLE]

which gives another upper bound of $1/\left(d+\bar{d}\right)$ on the update rate. Combining the two upper bounds gives the result in (41).  $\blacksquare$

The above lemma’s result is quite intuitive. It shows that there can be two bottlenecks to the system’s number of updates, one due to energy constraints, and the other due to service time constraints. Determining which one is in effect depends on the value of $d+\bar{d}$: if it is larger than $1$, which is the average energy arrival rate at both the source and the relay, then the system is more constrained by the service times; and if it is less than $1$, then the system is more constrained by the energy arrival rate. This simple observation leads to a variation in the optimal update policy as we show in the sequel. In the next lemma, we use the above result to derive a lower bound on the optimal solution, $\rho$, of problem (16). The proof of the lemma is in Appendix -D.

Lemma 7

The following lower bound holds:

[TABLE]

Observe that Lemma 7 reemphasizes the fact that the system’s bottleneck depends on the relationship between the total service time $d+\bar{d}$ and the average energy arrival rate. Clearly, if $d+\bar{d}<1$, the lower bound would be given by the first term in the maximum in (44), and would be given by the second term otherwise. We will use this fact while devising the optimal policy below. Let us now define the following online update policy:

Definition 2** **(Best Effort Uniform Update Policy with Service Constraints)

The source schedules transmission of a new status update at times: $\ell_{n}\triangleq n\cdot\max\left\{1,d+\bar{d}\right\}$, $n=0,1,2,\dots$. At $\ell_{n}^{-}$, if the source and the relay have at least $E$ and $\bar{E}$ units of energy, respectively, then the update is transmitted from the source and gets forwarded by the relay directly once it is received. Otherwise, if either the source or the relay has an empty battery at $\ell_{n}^{-}$, then both nodes stay silent until the next scheduled transmission time $\ell_{n+1}$.

Observe that the best effort uniform update policy with service constraints has a scheduled update rate of $1/\max\left\{1,d+\bar{d}\right\}=\min\left\{1,1/(d+\bar{d})\right\}$, which is the maximal possible update rate according to (41). Our goal now is to show that such rate is indeed achievable. This is mainly shown by proving that the failure update rate, which is given by the ratio of the number of scheduled update times in which no transmission occurs (due to either the source or the relay having an empty battery) to the total elapsed time, is negligible in the long run. This is formally proven in the next theorem, the main result of this section, whose proof is in Appendix -E.

Theorem 4

The long term average AoI under the best effort uniform update policy with service constraints achieves the lower bound in Lemma 7, and is therefore optimal.

We note that when $d+\bar{d}\geq 1$, the optimal policy is equivalent to a greedy policy, in which an update is sent whenever both the source and the relay have enough energy. This is clear from the fact that the best effort uniform update policy with service constraints schedules updates every $d+\bar{d}$ time units, i.e., back-to-back. The performance of the greedy policy, however, deteriorates when $d+\bar{d}<1$. We touch upon this note again in the numerical results section below.

We also note that, as in the offline setting, the optimal solution of the online setting is also extendible to general multihop scenarios with $M>1$ relays. This is attributed to the fact that the best effort uniform update policy with service constraints implicitly treats the source and the relay as one combined node, and only goes ahead with a transmission if both of them have enough energy simultaneously.

V Numerical Results

We now present some numerical examples to further illustrate our results. We start by the offline ones. A two-hop network has energy arriving at times ${\bm{s}}=[2,6,7,11,13]$ at the source, and $\bar{{\bm{s}}}=[1,4,9,10,15]$ at the relay. A source transmission takes $d=1$ time unit to reach the relay; a relay transmission takes $\bar{d}=2$ time units to reach the destination. Session time is $T=19$. We apply the change of parameters in Theorem 3 to get new energy arrival times ${\bm{s}}=[3,7,9,12,15]$, new transmission delay $d=3$, and new session time $T=20$. Then, we solve problem (III-A) to get the optimal inter-update times, using the new parameters. Note that $T\geq(N+1)d=18$, whence the optimal solution is given by Theorem 2. We apply the Inter-Update Balancing algorithm to get ${\bm{x}}^{e}=[6.5,6.5,5.67,5.67,5.67,5]$. Hence, the first infeasible inter-update time occurs at $n_{0}=3$ ($x_{3}^{e}<2d=6$). Thus, we set: $x_{1}^{*}=x_{1}^{e}$ and $x_{2}^{*}=x_{2}^{e}$; $x_{3}^{*}=x_{4}^{*}=x_{5}^{*}=2d$; and $x_{6}^{*}=T+Nd-\sum_{i=1}^{5}x_{i}^{*}$. We see that ${\bm{x}}^{*}=[6.5,6.5,6,6,6,4]$ satisfies the conditions stated in Lemmas 1, 2, and 3. In order to show how this translates back into the original two-hop network, in Fig. 6 we plot the achievable AoI curve at the destination, along with the optimal transmission times at both the source and the relay. We compute the optimal area under the AoI curve in this case to be $53.25$. We compare such area with the one achieved using an offline greedy policy, in which the source submits a new measurement whenever feasible, and the relay forwards it whenever possible. This is basically done by satisfying all the constraints of problem (II-A) with equality recursively as follows: $t_{1}=s_{1}$, $\bar{t}_{1}=\max\{\bar{s}_{1},s_{1}+d\}$, $t_{2}=\max\{s_{2},\bar{t}_{1}+\bar{d}\}$, $\bar{t}_{2}=\max\{\bar{s}_{2},t_{2}+d\}$, and so on. The achievable area under the AoI curve under such offline greedy policy is $53.5$, which is strictly higher than that achieved by the optimal solution.

We consider another example where energy arrives at times ${\bm{s}}=[0,4,4,9,13]$ and $\bar{{\bm{s}}}=[1,3,6,10,12]$, with $T=16$. Applying the change of parameters in Theorem 3 we get $T=17<(N+1)d=18$, and hence we use the results of Theorem 1 to get ${\bm{x}}^{*}=[5,6,6,6,6,3]$. The area under the AoI curve achieved with the optimal solution is given by $40$, while that achieved by the offline greedy policy is given by $42$. We then increase $T$ to $18$. This is effectively $19$ according to Theorem 3, and therefore we apply Theorem 2 results. The Inter-Update Balancing algorithm gives ${\bm{x}}^{e}=[5.8,5.8,5.8,5.8,5.8,5]$, and hence $n_{0}=2$. Since $x_{1}^{e}>s_{1}+d=4$, then the optimal solution is given by (26)-(28) as ${\bm{x}}^{*}=[5,6,6,6,6,5]$. As we compared before, the area under the AoI curve achieved with the optimal solution is given by $48$, while that achieved by the offline greedy policy is given by $50$.

We now present some online results. We create two independent sample paths according to unit-rate Poisson processes at the source and at the relay over the duration of $5\times 10^{3}$ time units. We then use them to compute the long term average AoI achieved by the proposed best effort uniform updating policy with service constraints as a function of the aggregate service time $d+\bar{d}$. We also compare this to a greedy policy, in which an update is transmitted whenever both the source and the relay have enough energy, regardless of the value of $d+\bar{d}$. We compute an average performance of both policies over $10^{3}$ iterations, and plot the results in Fig. 7. In the figure, we also plot the theoretical lower bound of Lemma 7. We see that the proposed policy and the lower bound are almost identical, as expected. We also see the superiority of the proposed policy in the low service time regime, i.e., when $d+\bar{d}<1$, compared to the greedy policy. In addition, as discussed after Theorem 4, the greedy policy becomes identical to the proposed policy in the high service time regime, when $d+\bar{d}\geq 1$.

In Fig. 8, we show how the time average AoI behaves as a function of $T$. We plot the achieved AoI under the best effort uniform update policy with service constraints and the greedy policy for values of $T$ ranging from $10$ to $10^{4}$ time units. In this example, $d+\bar{d}=0.25$ time units. We see from the figure that as $T$ grows, the greedy policy’s performance deteriorates, while the performance of the best effort uniform update policy with service constraints approaches the theoretical long term average lower bound derived in Lemma 7, confirming the results of Theorem 4. We note that the reason there is a small dip below the theoretical lower bound for small values of $T$ is that the bound in Lemma 7 is an asymptotic one that is effective only for relatively large values of $T$.

VI Conclusions and Future Directions

Age-minimal transmission policies have been proposed for energy harvesting two-hop networks with fixed service times. It has been shown that the optimal update policy is such that the relay’s data buffer should not contain any update packets waiting for service. Instead, updates should be transmitted from the source just in time as the relay is ready to forward them to the destination. In the offline setting, the optimal source transmission times, minimizing the average age of information by a given deadline, were found by an inter-update balancing algorithm that takes service times into consideration, as well as the knowledge of the incoming energy arrival information. In the online setting, a best effort uniform update policy with service constraints is shown to minimize the long term average age of information, in which time is uniformly divided into time slots of durations depending on service times, and an update is transmitted from the source at the beginning of each time slot only if both the source and the relay have enough energy.

The considered system model in the online setting of this paper is centralized, in which energy arrivals at both the source and the relay nodes are revealed simultaneously causally over time to some central controller that takes transmission decisions. It would be of interest to extend this work to a decentralized setting, in which both nodes take their own decisions independently of each other. Clearly, such decentralized problem is more challenging, but it may represent some practical applications where no central controller is present. Another line of extension would be to consider the finite battery case, in both centralized and decentralized scenarios, and extend the notions of threshold policies, which are optimal in single-hop settings [33, 34], to multihop settings.

-A Proof of Lemma 4

We show this by induction. Clearly, we have $\bar{x}_{1}=\bar{x}_{2}=\dots=\bar{x}_{i_{1}}=\frac{s_{i_{1}}}{i_{1}}+d$ by construction. Now assume that $\{\bar{x}_{i}\}_{i=1}^{i_{k}}$ is non-increasing, and consider $\{\bar{x}_{i}\}_{i=1}^{i_{k+1}}$. We know that $\bar{x}_{i_{k}+1}=\bar{x}_{i_{k}+2}=\dots=\bar{x}_{i_{k+1}}=\frac{s_{i_{k+1}}-s_{i_{k}}}{i_{k+1}-i_{k}}+d$ by construction. We now proceed by contradiction; assume that $\bar{x}_{i_{k+1}}>\bar{x}_{i_{k}}$. This means that the following holds:

[TABLE]

or equivalently

[TABLE]

Next, observe that the following holds by construction when choosing $x_{i_{k}}$:

[TABLE]

which is equivalent to

[TABLE]

This contradicts (46), and proves the first part of the lemma.

Now let us show the second part. Assume that $\bar{x}_{j+1}<\bar{x}_{j}$. Then necessarily we must have $j=i_{k}$ for some $i_{k}$, or else they should be equal. Therefore, by construction, we have

[TABLE]

This concludes the proof.

-B Proof of Lemma 5

Let $\{\bar{x}_{i}\}_{i=1}^{N}$ be the output of the Inter-Update Balancing algorithm and let $\{x_{i}^{e}\}_{i=1}^{N}$ denote the optimal solution of problem $(P^{e})$. We first show that $\{\bar{x}_{i}\}_{i=1}^{N}$ is feasible. Let $j$ be such that $i_{k}<j\leq i_{k+1}$. Then

[TABLE]

where $(\ref{eq_a})$ follows by (-A), and $(\ref{eq_b})$ follows since, by construction, we have

[TABLE]

Finally, note that the stopping criterion of the algorithm is when $i_{L}=N+1$ for some $i_{L}$. Whence, we have

[TABLE]

This shows that that $\{\bar{x}_{i}\}_{i=1}^{N}$ is feasible.

Next, we show that $x_{i}^{e}=\bar{x}_{i},~{}\forall i$. We show this by contradiction. Let $x_{i}^{e}=\bar{x}_{i}$ for $1\leq i\leq m-1$ and let $x_{m}^{e}\neq\bar{x}_{m}$, i.e., $m$ is the first time index at which the two sequences are different. We now consider two cases as follows.

First, assume $x_{m}^{e}<\bar{x}_{m}$. Note that it must be the case that $i_{k_{1}}<m\leq i_{k}$ for some $i_{k}$. Therefore, $\bar{x}_{i}=\bar{x}_{m},~{}\forall m\leq i\leq i_{k}$, by construction. By (21), we have that $\{x_{i}^{e}\}$ is non-increasing since $\eta_{i}=0$ and $\lambda_{i}\geq 0$. Therefore, $x_{i}^{e}<\bar{x}_{i},~{}\forall m\leq i\leq i_{k}$, and hence $\sum_{i=1}^{i_{k}}x_{i}^{e}<\sum_{i=1}^{i_{k}}\bar{x}_{i}=s_{i_{k}}+i_{k}d$, i.e., the allegedly-optimal policy is not feasible. Therefore $x_{m}^{e}\geq\bar{x}_{m}$.

Second, assume $x_{m}^{e}>\bar{x}_{m}$. Since $\sum_{i=1}^{N+1}x_{i}^{e}=\sum_{i=1}^{N+1}\bar{x}_{i}$, therefore there must exist some time index $l>m$ such that $x_{l}^{e}<\bar{x}_{l}$. Now let $\epsilon\triangleq\min\{x_{m}^{e}-\bar{x}_{m},\bar{x}_{l}-x_{l}^{e}\}$, and consider a new policy $\{\tilde{x}_{i}\}$ which is equal to $\{x_{i}^{e}\}$ except at time indices $m$ and $l$, with $\tilde{x}_{m}=x_{m}^{e}-\epsilon$ and $\tilde{x}_{l}=x_{l}^{e}+\epsilon$. Since $\tilde{x}_{m}\geq\bar{x}_{m}$, the new policy is feasible. In addition, by convexity of the square function, the following holds [42]:

[TABLE]

which means that the new policy achieves a lower age, rendering $\{x_{i}^{e}\}$ suboptimal.

The above arguments show that we must have $x_{i}^{e}=\bar{x}_{i},~{}\forall i$. This completes the proof.

-C Proof of Theorem 2

The first part of the theorem follows directly since the solution of the less constrained problem $\left(P^{e}\right)$ is optimal if feasible in problem (III-A). Next, we prove the second part.

We first show that $n_{0}\leq N$ by contradiction. Assume that $n_{0}=N+1$, i.e., $x_{N+1}^{e}<d$ and $x_{N}^{e}\geq 2d>x_{N+1}^{e}$. By Lemma 4, this means that $\sum_{i=1}^{N}x_{i}^{e}=s_{N}+Nd$. Hence, $x_{N+1}^{e}=T+Nd-s_{N}-Nd=T-s_{N}$, which cannot be less than $d$ by the feasibility assumption in (10). Thus, $n_{0}\leq N$.

Now let $n_{0}>2$ and observe that $x_{n_{0}}^{e}<2d\leq x_{n_{0}-1}$. Thus, by Lemma 4, we must have $\sum_{i=1}^{n_{0}-1}x_{i}^{e}=s_{n_{0}-1}+(n_{0}-1)d$. Now let us show that the proposed policy is feasible; we only need to check whether $x_{N+1}^{*}\geq d$. Towards that, we have

[TABLE]

where the last inequality follows by the feasibility assumption in (10). Therefore, the proposed policy is feasible.

We now show that it is optimal as follows. Assume that there exists another policy $\{\tilde{x}_{i}\}$ that achieves a lower age than $\{x_{i}^{*}\}$. We now have two cases. First, assume that $\sum_{i=1}^{n_{0}-1}\tilde{x}_{i}=s_{n_{0}-1}+(n_{0}-1)d$. then by Corollary 1 we must have $\tilde{x}_{i}=x_{i}^{*}$ for $1\leq i\leq n_{0}-1$. Now for $n_{0}\leq i\leq N$, if $\tilde{x}_{i}>x_{i}^{*}=2d$, this means that $\tilde{x}_{N+1}<x_{N+1}^{*}$ to satisfy the last constraint in (III-A). Since $\sum_{i=n_{0}}^{N+1}\tilde{x}_{i}=\sum_{i=n_{0}}^{N+1}x_{i}^{*}$, then by convexity of the square function, $\sum_{i=n_{0}}^{N+1}\left(\tilde{x}_{i}\right)^{2}>\sum_{i=n_{0}}^{N+1}\left(x_{i}^{*}\right)^{2}$ [42], and hence $\{\tilde{x}_{i}\}$ cannot be optimal. Second, assume that $\sum_{i=1}^{n_{0}-1}\tilde{x}_{i}>s_{n_{0}-1}+(n_{0}-1)d=\sum_{i=1}^{n_{0}-1}x_{i}^{*}$. Since $\tilde{x}_{i}\geq x_{i}^{*}=2d$ for $n_{0}\leq i\leq N$, and $\sum_{i=1}^{N+1}\tilde{x}_{i}=\sum_{i=1}^{N+1}x_{i}^{*}$, then we must have $\tilde{x}_{N+1}<x_{N+1}^{*}$. Thus, $\sum_{i=1}^{N+1}\left(\tilde{x}_{i}\right)^{2}>\sum_{i=1}^{N+1}\left(x_{i}^{*}\right)^{2}$ by convexity of the square function [42], and $\{\tilde{x}_{i}\}$ cannot be optimal.

Finally, let $n_{0}=2$. If $x_{1}^{e}=s_{1}+d$, then the proof follows by the arguments for the $n_{0}>2$ case. Else if $x_{1}^{e}>s_{1}+d$, then $x_{1}^{e}=x_{2}^{e}\geq x_{N+1}^{e}$ by Lemma 4. Since $\{x_{i}^{e}\}_{i=2}^{N}$ have to increase to at least $2d$, then $x_{1}^{e}+x_{N+1}^{e}$ has to decrease to satisfy the last constraint in (III-A). However, one cannot increase $x_{1}^{e}$ to $2d$ or more and compensate that by decreasing $x_{N+1}^{e}$, by convexity of the square function. Thus, $x_{1}^{*}<x_{2}^{*}$, and Lemma 2 shows that the results of Theorem 1 follow to give (26)-(28).

-D Proof of Lemma 7

We start by noting that, as observed in the proof of Theorem 3, the objective function is minimized if the data causality constraints hold with equality at all times. Ignoring the remainder of the constraints, we have that

[TABLE]

where (58) follows by ignoring the last term in (57) and using Fatou’s lemma; (59) follows by Jensen’s inequality (convexity of the square function) and using the upper bound in (41); and (61) follows by considering a policy that achieves the bound in (41). The result now follows after simple manipulations of the last equation above.

-E Proof of Theorem 4

We show this by analyzing the performance of a slightly different update policy that cannot perform better than the proposed best effort uniform update policy with service constraints, and proving that it achieves the lower bound in Lemma 7. Such policy is different from the proposed policy in that whenever the source or the relay have an empty battery at some $\ell_{n}^{-}$, then not only they both stay silent until the next scheduled update time $\ell_{n+1}$, but whichever node has a non-zero amount of energy in its battery at $\ell_{n}^{-}$ dumps that amount. Despite the fact that one node dumping its available energy when the other node does not have any seems quite wasteful, it is considered mainly to facilitate the analysis. We now proceed with that dumping condition in action.

We adopt the proof techniques involved in proving the optimality of best effort uniform updating policies for single-user channels with zero service times studied in [25], after altering them to fit for our two-hop network with non-zero service times. Specifically, let $u_{i}$ denote the length of the $i$th interval of time that starts with an empty battery at either the source or the relay at time $\ell_{n_{i}}^{-}$ for some $n_{i}\geq 1$, i.e., either $\mathcal{E}\left(\ell_{n_{i}}^{-}\right)=0$ or $\bar{\mathcal{E}}\left(\ell_{n_{i}}^{-}\right)=0$, and ends with both nodes having sufficient energy units to transmit an update at time $\ell_{n_{i}+u_{i}}^{-}$, i.e., with $\mathcal{E}\left(\ell_{n_{i}+u_{i}}^{-}\right)\geq E$ and $\bar{\mathcal{E}}\left(\ell_{n_{i}+u_{i}}^{-}\right)\geq\bar{E}$. Similarly, let $v_{j}$ denote the length of the $j$th interval of time that starts with both the source and the relay having sufficient energy to transmit an update at time $\ell_{n_{j}}^{-}$ for some $n_{j}\geq 1$, and ends when at least one of the two nodes have an empty battery at time $\ell_{n_{j}+v_{j}}^{-}$. With the initial condition $\mathcal{E}=E$ and $\bar{\mathcal{E}}=\bar{E}$, the source and the relay start with sending an update at time [math]. One can then group the scheduled update time slots into $v_{1},u_{1},v_{2},u_{2},\dots$, see Fig. 9 for a visual example of the above notation. Observe that the shaded area in the figure, representing the area under the age curve between times [math] and $\ell_{1}$ does not affect the long term average AoI as it is merely a finite area.

We are interested in the distribution of the $u_{i}$’s, as it will prove beneficial later in the proof. Let $A$ and $\bar{A}$ be two random variables denoting the number of energy arrivals harvested in between two scheduled update times at the source and the relay, respectively. Since the energy arrival processes at the source and the relay are i.i.d. Poisson with unit rate, it then follows that both $A$ and $\bar{A}$ are i.i.d. Poisson random variables with parameter $\max\left\{1,d+\bar{d}\right\}$. According to the policy, both nodes start with empty batteries at $\ell_{n_{i}}$ (whichever node has a non-zero amount of energy at $\ell_{n_{i}}^{-}$ dumps that amount). Therefore, it holds that333Without that dumping condition, analyzing the distribution of the $u_{i}$’s becomes more complicated, as opposed to the quite simplistic current form.

[TABLE]

where $p\triangleq 1-1/e^{\max\left\{1,d+\bar{d}\right\}}$, and the first equality follows from the fact that according to the definition of the update policy, whenever a scheduled update time fails, both nodes reset their batteries. Thus, the only way to have a successful scheduled update after a failed one is that at least one energy arrival occurs at both nodes. The second equality follows by independence of $A$ and $\bar{A}$. Therefore, $u_{i}$’s are i.i.d. geometrically distributed random variables with parameter $p^{2}$, with finite first and second moments. This latter fact will come helpful later in the proof.

Let $K(T)$ denote the number of failed scheduled updates within time $T$. Now let us pick $T$ to be an integer multiple of $\max\left\{1,d+\bar{d}\right\}$, and pick it larger than $\sum_{i=1}^{K(T)}u_{i}$. Therefore, it holds that

[TABLE]

We now show that the term $\frac{1}{T}\sum_{i=1}^{K(T)}u_{i}$, representing the ratio of time in which no updates are transmitted, goes to [math] a.s. as $T$ grows infinitely large. Toward that end, we first argue that $K(T)\rightarrow\infty$ a.s., which implies by the strong law of large numbers (SLLN), it holds that $\frac{1}{K(T)}\sum_{i=1}^{K(T)}u_{i}\rightarrow\mathbb{E}\left[u_{i}\right]$ a.s., the first moment of a geometric random variable with parameter $p^{2}$, which is finite. We then argue that $K(T)/T\rightarrow 0$ a.s., rendering $\frac{K(T)}{T}\frac{1}{K(T)}\sum_{i=1}^{K(T)}u_{i}\rightarrow 0$ a.s. as required. Note that such result implies that the update rate $N(T)/T$ under the best effort uniform update policy with service constraints converges to the upper bound in (41), which is a necessary step toward showing the achievability of the lower bound on the AoI in Lemma 7. The fact that $K(T)\rightarrow\infty$ and $K(T)/T\rightarrow 0$ can be shown along the same lines of the proof of [43, Theorem 1] (see also the proof of [25, Theorem 1]). We highlight the main difference in what follows. In [43, Appendix A], the amount of energy in the sensor’s battery starting at the beginning of an interval $v_{j}$ until its end is represented by a random walk. It is then shown that such random walk reaches [math] within a finite time a.s. through a series of indirect steps involving properties of martingales and Poisson processes. Once this is shown, the result follows. In our setting, one can represent the amounts of energy in the source’s and the relay’s batteries as two i.i.d. random walks, each of which having a finite [math]-hitting time a.s. Since our proposed policy stops if either of the two nodes have an empty battery, i.e., at the minimum of both [math]-hitting times, then this will occur in a finite time a.s. as well. Since, according to our proposed policy, both nodes will have [math] energy at such hitting time (recall that whichever node has an extra amount of energy it gets rid of it), the rest of the proof in [43, Appendix A] directly follows.

Next, let us define the following function denoting the area of the trapezoids that constitute $A_{T}$:

[TABLE]

where $x$ denotes the side length of the trapezoid that is on the horizontal axis. Therefore, one can write

[TABLE]

Now observe that by SLLN the quantities $\frac{1}{K(T)}\sum_{i=1}^{\infty}u_{i}$ and $\frac{1}{K(T)}\sum_{i=1}^{\infty}u_{i}^{2}$ converge, a.s., to the first and second moments of a geometric random variable with parameter $p^{2}$, respectively, which are both finite quantities. Since $K(T)/T\rightarrow 0$ as $T\rightarrow\infty$, it therefore follows that under the best effort uniform update policy with service constraints, the following long term average AoI is achievable:

[TABLE]

This completes the proof.