Trajectory Optimization for Rotary-Wing UAVs in Wireless Networks with Random Requests

TL;DR

This paper presents a novel trajectory optimization method for rotary-wing UAVs acting as data relays in wireless networks with randomly generated requests, aiming to minimize average communication delay.

Contribution

It introduces a two-scale optimization approach for UAV trajectory planning using semi-Markov decision processes, improving delay performance over heuristics.

Findings

Optimal UAV movement towards the geometric center reduces delay.

The two-scale optimization significantly outperforms simple heuristics.

End positions become payload-independent at large data sizes.

Abstract

This paper studies the trajectory optimization problem in a scenario where a single rotary-wing UAV acts as a relay of data payloads for downlink transmission requests generated randomly by two ground nodes (GNs) in a wireless network. The goal is to optimize the UAV trajectory in order to minimize the expected average communication delay to serve these random requests. It is shown that the problem can be cast as a semi-Markov decision process (SMDP), and the resulting minimization problem is solved via multi-chain policy iteration. The optimality of a two-scale optimization approach is proved: the optimal trajectory in the communication phase greedily minimizes the communication delay of the current request while moving between the current start position and a target end position (inner optimization); the end positions are selected to minimize the expected average long-term delay in the SMDP (outer optimization). Numerical simulations show that the expected average delay is minimized when the UAV moves towards the geometric center of the GNs during phases in which it is not actively servicing transmission requests, and demonstrate significant improvements over sensible heuristics. Finally, it is revealed that the optimal end positions of communication phases become increasingly independent of the data payload, for large data payload values.

Equations

$$R_{r}(q(t))=B\log_{2}\Bigg{(}1+\frac{\gamma}{H^{2}+(q(t)-x_{r})^{2}}\Bigg{)},$$

$$\bar{D}_{\mu}=\lim_{t\rightarrow\infty}\mathbb{E}\Bigg{[}\frac{\sum_{l=0}^{M_{t}-1}\Delta_{l}}{M_{t}}\Bigg{|}q(0)=0\Bigg{]}.$$

$${\mu }^{\ast }=\arg \underset{\mu }{\min }{\bar{D}}_{\mu }.$$

$$\mathcal{I}\triangleq \left\{-N,N+1,\dots ,N-1,N\right\}$$

$${\Delta }_0\triangleq \frac{a}{NV}.$$

$$\mathbb{P}(s_{n+1}=(i+m,0)|s_n=(i,0),a_n=m)=e^{-\lambda {\Delta }_0},$$

$$\mathbb{P}(s_{n+1}=(i+m,r)|s_n=(i,0),a_n=m)=\frac{1-e^{-\lambda {\Delta }_0}}{2},\forall r\in \{1,2\},$$

$${\int }_0^{\Delta }{R}_r(q(\tau ))d\tau \ge L,$$

$$\mathcal{T}_{r}(i\to j)=\biggr{\{}q:[0,\Delta]\to[-a,a]:\int_{0}^{\Delta}R_{r}(q(\tau))d\tau\geq L,$$

$$|q^{\prime}(t)|\leq V,\ q(0)={q}_{i},\ q(\Delta)={q}_{j},\ \exists\Delta>0\biggr{\}}.$$

$$\mathbb{P}(s_{n+1}=(j,0)|s_n=(i,r),q)=1,\forall q\in {\mathcal{T}}_r(i\to j).$$

$$\bar{D}_{\mu}{=}\lim_{K\rightarrow\infty}\mathbb{E}\Bigg{[}\frac{\frac{1}{K}\sum_{n=0}^{K-1}\Delta_{s_{n}}(\mu(s_{n}))}{\frac{1}{K}\sum_{n=0}^{K-1}\chi(s_{n}\in\mathcal{S}_{\mathrm{comm}})}\Bigg{|}s_{0}=(0,0)\Bigg{]},$$

$${\bar{D}}_{\mu }$$

$$=\frac{{\sum }_{s\in {\mathcal{S}}_{\mathrm{comm}}}{\Pi }_{\mu }(s){\Delta }_s(\mu (s))}{{\sum }_{s\in {\mathcal{S}}_{\mathrm{comm}}}{\Pi }_{\mu }(s)},$$

$${\pi }_{\mathrm{wait}}=\frac{1}{2-e^{-\lambda {\Delta }_0}},{\pi }_{\mathrm{comm}}=\frac{1-e^{-\lambda {\Delta }_0}}{2-e^{-\lambda {\Delta }_0}}.$$

$${\pi }_{\mathrm{wait}}=p_{ww}{\pi }_{\mathrm{wait}}+p_{cw}{\pi }_{\mathrm{comm}}=e^{-\lambda {\Delta }_0}{\pi }_{\mathrm{wait}}+{\pi }_{\mathrm{comm}},$$

$${\pi }_{\mathrm{comm}}=p_{wc}{\pi }_{\mathrm{wait}}+p_{cc}{\pi }_{\mathrm{comm}}=(1-e^{-\lambda {\Delta }_0}){\pi }_{\mathrm{wait}},$$

$${\pi }_{\mathrm{wait}}+{\pi }_{\mathrm{comm}}=1,$$

$${\bar{D}}_{\mu }=\frac{{\sum }_{s\in \mathcal{S}}{\Pi }_{\mu }(s){\Delta }_s(\mu (s))}{{\pi }_{\mathrm{comm}}},$$

$${\bar{D}}_{\mu }^{\ast }=\frac{\underset{\theta ,J}{\min }\sum _{s\in {\mathcal{S}}_{\mathrm{comm}}}{\Pi }_{\theta ,J}(s)\underset{\rho (s,J(s)))}{\min }{\Delta }_s(J(s),\rho (s,J(s)))}{{\pi }_{\mathrm{comm}}}.$$

$${\Delta }_r^{\ast }(i,j)\triangleq \underset{q(\cdot )\in {\mathcal{T}}_r(i\to j)}{\min }{\Delta }_{i,r}(j,q),\forall (i,r)\in {\mathcal{S}}_{\mathrm{comm}},\forall j\in \mathcal{I},$$

$${\bar{D}}_{\mu }^{\ast }=\frac{\underset{\theta ,J}{\min }{\sum }_{(i,r)\in {\mathcal{S}}_{\mathrm{comm}}}{\Pi }_{\theta ,J}(i,r){\Delta }_r^{\ast }(i,J(i,r))}{{\pi }_{\mathrm{comm}}}.$$

$${\Delta }_r^{\ast }(i,j)=\underset{\Delta ,q}{\min }$$

$$\displaystyle|q^{\prime}(\tau)|\leq V,q(0)=q_{i},q(\Delta)=q_{j}\Big{\}}.$$

$${\ell }_{p_1,p_2}^{(r)}\triangleq {\int }_0^{{\tau }_{p_1,p_2}}{R}_r\left(p_1+\frac{\tau }{{\tau }_{p_1,p_2}}(p_2-p_1)\right)d\tau$$

$$\upsilon \{p_1\to (p_2,\delta )\to p_3\}(\tau )$$

$$=\{\begin{array}{ll}{p}_1+\frac{\tau }{{\tau }_{p_1,p_2}}(p_2-p_1),& \tau \in [0,{\tau }_{p_1,p_2}]\\ p_2,& \tau \in [{\tau }_{p_1,p_2},{\tau }_{p_1,p_2}+\delta ]\\ p_2+\frac{\tau -{\tau }_{p_1,p_2}-\delta }{{\tau }_{p_2,p_3}}(p_3-p_2),& \tau \in [{\tau }_{p_1,p_2}+\delta ,{\tau }_{p_1,p_2}+{\tau }_{p_2,p_3}+\delta ].\end{array}$$

$$q^{\ast }(\cdot )=\upsilon \{q_i\to (q_j,0)\to q_j\}(\cdot ),{\Delta }_r^{\ast }(i,j)={\tau }_{q_i,q_j},$$

$$q^{\ast }(\cdot )=\upsilon \{q_i\to (x_r,{\delta }^{\ast })\to q_j\}(\cdot ),{\Delta }_r^{\ast }(i,j)={\tau }_{q_i,x_r}+{\tau }_{x_r,q_j}+{\delta }^{\ast },$$

$${\delta }^{\ast }=\frac{L-{\ell }_{q_i,x_r}^{(r)}-{\ell }_{x_r,q_j}^{(r)}}{R_r(x_r)};$$

$$q^{\ast }(\cdot )=\upsilon \{q_i\to (p^{\ast },0)\to q_j\}(\cdot ),{\Delta }_r^{\ast }(i,j)={\tau }_{q_i,p^{\ast }}+{\tau }_{p^{\ast },q_j},$$

$$\theta^{*}(i){=}\left\{\begin{array}[]{ll}1,&i\in\{-N,-N+1,...,-1\}\\ 0,&i=0\\ -1,&i\in\{1,2,...,N\}.\end{array}\right.$$

$${\bar{D}}_{\mu }^{\ast }\approx \frac{\underset{\theta ,J}{\min }{\sum }_{(i,r)\in {\mathcal{S}}_{\mathrm{comm}}}{\Pi }_{\theta ,J}(i,r)L}{{\pi }_{\mathrm{comm}}B{\log }_2(1+\gamma /H^2)}=\frac{L}{B{\log }_2(1+\gamma /H^2)}.$$

Full text

I Introduction

Recently, much research has gone into UAVs operating in wireless networks [1, 2, 3, 4, 5]. The drive for this is due to the unique benefits that UAVs acting as flying base stations, mobile relays, etc., provide in enhancing the overall network performance, thanks to their unique advantages over terrestrial counterparts in terms of mobility, maneuverability, and improved line-of-sight (LoS) link probability [1]. However, the design of UAV deployment strategies comes with challenges, namely the determination of optimal positioning or trajectories in the face of constraints imposed on UAV energy consumption, network throughput, and/or delay requirements [1, 2, 3, 4].

Some research has focused on the trajectory optimization under energy constraints, as in [2] and [3]. In [6], the fine-grained structure of LoS conditions is exploited to position UAVs optimally with the goal to maximize throughput. In [4], a model-free Q-learning approach is taken in the trajectory design so as to maximize the transmission sum-rate.

All of these efforts consider situations that are solved in the offline case, i.e., the pattern of transmission requests is known in advance, so that the trajectory may be pre-planned accordingly. However, this may be impractical as transmission requests are often random and cannot be determined in advance. In these cases, trajectory design is much more challenging, since it must be continuously adjusted based on the realization of these random processes, and incorporate the uncertainty in the future evolution of the system dynamics. In this paper, we investigate this problem by developing policies that adapt the trajectory based on the random realization of downlink transmission requests generated by two ground nodes (GNs), so as to optimize the average long-term performance.

To further motivate the need for this new formulation, consider the scenario depicted in Fig. 1. In this context, the minimum communication delay to serve GN1 is achieved by flying towards it to improve the distance-dependent pathloss. With this design, for a sufficiently large data payload, the UAV will terminate the data transmission hovering above GN1, where channel conditions are most favorable. However, if the UAV is to service a random request generated by GN2 shortly after terminating the transmission to GN1, the delay incurred to serve this second request may be large due to the large distance that separates the UAV from GN2, causing severe pathloss conditions. In other words, under random transmission requests, the greedy delay minimization to serve a certain request may lead the UAV to a position where subsequent random requests cannot be served effectively, yielding poor delay performance in an average long-term sense. This example points to the need to incorporate the random nature of transmission requests in the trajectory design.

To address this question, we consider a scenario in which an UAV is serving two GNs far apart, and receives transmission requests according to a Poisson random process. We formulate the problem as that of designing an adaptive trajectory, so as to minimize the average long-term communication delay incurred to serve the requests of both GNs. We prove that the optimal trajectory in the communication phase operates according to a two-scale optimization: in the outer optimization, the UAV selects a target end position, which optimizes the trade-off between minimizing the delay of the current request, and minimizing the expected average long-term delay; then, in the inner optimization, the UAV travels greedily from the current position to the selected end position while communicating, following the trajectory that greedily minimizes the communication delay for the current request, provided in closed form. We utilize a multi-chain policy iteration algorithm to optimize the selection of the end position in the communication phase and the trajectory during the waiting phase, in which the UAV is not actively servicing downlink transmission requests. Our numerical results reveal that the UAV should always move towards the geometric center of the two GNs during the waiting phase, and that the optimal trajectory during communication phases becomes increasingly independent of the data payload and only determined by system parameters as the data payload value becomes sufficiently large.

The rest of the paper is organized as follows. In Sec. II, we introduce the system model and state the optimization problem; in Sec. III, we formalize the problem as a semi-Markov decision process (SMDP); in Sec. IV, we provide numerical results; lastly, in Sec. V, we conclude the paper with some final remarks.

II System Model and Problem Formulation

II-A System Model

Consider the scenario depicted in Fig. 1, where one rotary-wing UAV services two ground nodes (GNs) with random downlink111This formulation and the analysis can be directly applied to uplink transmissions as well. transmission requests of $L$ bits. The two ground units GN1 and GN2 are located at positions $x_{1}{=}-a$ and $x_{2}{=}a$ along the $x$-axis, respectively, both at ground level (height [math]). The UAV moves along the line segment connecting the two GNs, at height $H$ from the ground. We let $q(t){\in}[-a,a]$ be the UAV’s position along the $x$-axis at time $t$, and we assume that it is either hovering or moving at speed $V$, hence $|q^{\prime}(t)|{\in}\{0,V\}$, where $f^{\prime}$ denotes derivative of $f$ over time.

A base station (BS) connected to the rest of the network is the source of downlink traffic to the two GNs. When a downlink request is generated by a certain GN, the BS transmits the data payload to the UAV, which then relays it to the GN using a decode and forward strategy [7]. We assume that the UAV has a high-capacity link to the BS, hence the communication link between the UAV and the GN constitutes the bottleneck of the overall BS-UAV-GN communication. In the rest of the paper, we thus focus on the UAV-GN communication and neglect the delay over the BS-UAV link. We assume that the UAV transmits at fixed power $P_{c}$ and that the communication intervals experience LoS links with no probabilistic elements. This is motivated by the fact that UAVs in low-altitude platforms generally tend to have a much higher occurrence of LoS links [8]. We model the instantaneous communication rate between the UAV in position $q(t)$ and GN${}_{r},r\in\{1,2\}$ in position $x_{r}$ as

[TABLE]

where $H^{2}+(q(t)-x_{r})^{2}$ is the squared distance between the UAV and GNr, $B$ is the channel bandwidth, and $\gamma$ is the SNR referenced at $1$ meter (see [3]).

When the UAV has no active transmission requests, future requests arrive according to a Poisson process with mean $\lambda/2$ requests/second, independently at each GN. Each request requires the transmission of $L$ bits to the corresponding destination. Upon receiving a request from GNr, the UAV enters the communication phase, where it services it by transmitting the $L$ bits to GNr; any additional requests received during this communication interval are dropped (see also Fig. 1). After the data transmission is completed, the UAV enters the waiting phase, where it awaits for new requests (with rate $\lambda/2$ for each GN), and the process is repeated indefinitely. During this periodic process of communication and waiting for new requests, the UAV follows a trajectory, part of our design, with the goal to minimize the average long-term communication delay, as discussed next.

II-B Problem Formulation

In this work, we consider the unconstrained delay minimization and neglect the propulsion energy consumption from our problem. In fact, it has been shown that a rotary-wing UAV exhibits comparable energy consumption when either moving or hovering [3]; in the special case when the moving and hovering powers are equal (for instance, based on the model in [3], this occurs at speed $V{=}38$ m/s), the finite energy in the UAV battery translates into a constraint on the total service time of the UAV, independent of the trajectory followed.

The goal is to define the optimal policy (UAV trajectory) so as to minimize the average communication delay. To this end, let $\Delta_{l}$ be the delay incurred to complete the transmission of the $l$th request serviced by the UAV. Let $M_{t}$ be the total number of requests served and completed up to time $t$. Then, we define the expected average delay under a given trajectory policy $\mu$ (to be defined), starting from $q(0)=0$ as222While in practice the operation time of the UAV is constrained by the amount of energy stored in its battery, and the policy should depend on the amount of time left, the asymptotic case $t\to\infty$ is convenient since it gives rise to stationary policies (i.e., time-independent); this is a good approximation when the dynamics of the waiting and communication phases occur at much faster time scales than the total travel time, i.e., when $M_{t}$ in (2) is large for practical values of the travel time $t$. For perspective, [9] places typical rotary-wing hovering endurance times in the 15-30 minute range.

[TABLE]

We then seek to determine $\mu^{*}$ to minimize $\bar{D}_{\mu}$, i.e.,

[TABLE]

Note that this is a non-trivial optimization problem. While the minimum delay to serve a request, say from GN1, is achieved by flying towards GN1 at maximum speed to improve the link quality, this strategy may not be optimal in an average delay sense: if the UAV receives a new request from GN2 shortly after completing the request to GN1, the delay to serve this second request may be large due to the large distance that must be covered by the UAV.

II-C Semi-Markov Decision Process (SMDP) formulation

In general, a solution to (3) would involve the optimization of an intractable number of variables over time (i.e., all possible trajectories followed by the UAV at any given time), over a continuous state space (the interval $[-a,a]$). Therefore, it is advantageous to approximate the system model through discretization and reformulate (3) as an average-cost SMDP.

We define the state space as $\mathcal{S}{=}\mathcal{I}{\times}\mathcal{R}$, where $\mathcal{R}{=}\{0,1,2\}$ denotes the request status, i.e., no active request ([math]), or a request is received from GNr ($r\in\{1,2\}$), and

[TABLE]

is the set of $2N+1$ indices corresponding to discretized positions $\mathcal{Q}\triangleq\{q_{i}=\frac{i}{N}a,\ \forall i\in\mathcal{I}\}$ along the interval $q(t)\in[-a,a]$. This is a good approximation for sufficiently large $N$, as $\frac{a}{NV}\lambda\ll 1$, making the expected number of requests received over the travel time between two adjacent discretized positions much smaller than one. It is also useful to further partition the state space into waiting states, $\mathcal{S}_{\mathrm{wait}}=\mathcal{I}\times\{0\}$, and communication states, $\mathcal{S}_{\mathrm{comm}}=\mathcal{I}\times\{1,2\}$.

To define this SMDP, we sample the continuous time interval to define a discrete sequence of states $\{s_{n},n\geq 0\}\subseteq\mathcal{S}$ with the Markov property. We now define the actions available, the transition probabilities, duration and cost of each state visit.

If the UAV is in state $s_{n}{=}(i,0){\in}\mathcal{S}_{\mathrm{wait}}$ at time $t$, i.e., it is in the discretized position $q_{i}$ and there are no active requests, then the actions available are $a_{n}{\in}\{-1,0,1\}$, i.e. move right ($a_{n}{=}1$ to position $q_{i{+}1}$), hover ($a_{n}{=}0$), or move left by one discretized position ($a_{n}{=}{-}1$ to $q_{i{-}1}$). The amount of time required to take this action, i.e., to fly between two adjacent discretized positions, is

[TABLE]

The new state is then sampled at time $t{+}\Delta_{0}$, and is given by $s_{n+1}{=}(i{+}a_{n},r_{n+1})$. The transition probability from state $s_{n}{=}(i,0)$ under action $a_{n}\in\{-1,0,1\}$ is then given by

[TABLE]

depending on whether no request is received during this time interval ($r_{n+1}{=}0$, with probability $e^{-\lambda\Delta_{0}}$), or a request is received from GNr ($r_{n+1}{=}r\in\{1,2\}$, with probability $[1-e^{-\lambda\Delta_{0}}]/2$ for each GN).

Upon reaching state $s_{n}{=}(i,r){\in}\mathcal{S}_{\mathrm{comm}}$ with $r{\in}\{1,2\}$ at time $t$, the UAV has received a request to serve $L$ bits to GNr. The actions available at this point are all trajectories that start from $q_{i}$ and allow the UAV to transmit the entire data payload of $L$ bits. Assuming a move and transmit strategy (see [3]), the selected trajectory $q(\cdot)$ of duration $\Delta$ must satisfy

[TABLE]

since all bits need to be transmitted during this phase. Under this trajectory, the communication delay is thus $\Delta$. We define the action space in state $(i,r)\in\mathcal{S}_{\mathrm{comm}}$ as the set of all feasible trajectories, $\mathcal{T}_{r}(i)=\cup_{j}\mathcal{T}_{r}(i\to j)$, where we have defined $\mathcal{T}_{r}(i\to j)$ as the set of feasible trajectories starting in ${q}_{i}$, ending in ${q}_{j}$, and serving GNr, i.e.,

[TABLE]

[TABLE]

Upon completing the communication phase, the UAV enters the waiting phase again; the new state is then sampled at time $t{+}\Delta$ (the amount of time elapsed to complete the selected trajectory), and is given by $s_{n+1}{=}(j,0){\in}\mathcal{S}_{\mathrm{wait}}$, corresponding to the position ${q}_{j}$ reached at the end of the communication phase. Thus, we have defined the transition probability in the SMDP from state $s_{n}=(i,r)$ under action $q\in\mathcal{T}_{r}(i\to j)$ as

[TABLE]

In other words, the trajectory selection process in the communication phase can be described via a two-scale decision process: 1) given $(i,r)$, i.e., the current position $q_{i}$ of the UAV and the request received from GNr, the UAV first selects some $j\in\mathcal{I}$, which defines the target position $q_{j}$ to be reached at the end of the communication phase; 2) the UAV selects a feasible trajectory $q$ from $\mathcal{T}_{r}(i\to j)$, executes the trajectory while communicating to GNr, and terminates the communication phase in the new position $q_{j}$, corresponding to state $(j,0)$. After this point, the UAV is in the waiting phase again.

With the states and actions defined, we can define a policy $\mu$. Specifically, for states $(i,0)\in\mathcal{S}_{\mathrm{wait}}$, $\mu(i,0)\in\{-1,0,1\}$. Likewise, for states $(i,r)\in\mathcal{S}_{\mathrm{comm}}$, $\mu(i,r)=(j,q(\cdot))$, where $j\in\mathcal{I}$ (position reached at the end of the communication phase) and $q(\cdot)\in\mathcal{T}_{r}(i\to j)$ (feasible trajectory starting in $q_{i}$, ending in $q_{j}$, to serve GNr).

The communication delay cost during the waiting phase is zero, i.e. $\Delta_{i,0}(m)=0$, for all states $(i,0)\in\mathcal{S}_{\mathrm{wait}}$ and actions $m\in\{-1,0,1\}$. When the UAV is in a communicating phase, we denote the communication delay incurred in state $(i,r)$ under action $(j,q(\cdot))$ as $\Delta_{i,r}(j,q(\cdot))$. Compactly, we write $\Delta_{s}(\mu(s))$ to denote the delay incurred in state $s\in\mathcal{S}$ under the action $\mu(s)$ dictated by policy $\mu$.

With this notation, and having now defined a stationary policy $\mu$, we can rewrite the average delay $\bar{D}_{\mu}$ in (2) in the context of the SMDP as

[TABLE]

where $\chi(A)$ is the indicator function of the event $A$. In fact, the numerator in (2) counts the sample average delay incurred in the communication phases up to slot $K$ of the SMDP, whereas the denominator in (2) counts the sample average number of communication slots in the SMDP up to slot $K$. Now, using Little’s Theorem [10], we can rewrite (10) as

[TABLE]

where $\Pi_{\mu}(s)$ is the steady-state probability in the SMDP of the UAV being in state $s$ under policy $\mu$, and the second equality holds since $\Delta_{s}(\mu(s))=0$ and $\chi(s{\in}\mathcal{S}_{\mathrm{comm}})=0$ for $s\in\mathcal{S}_{\mathrm{wait}}$.

III Policy Optimization and Analysis

In this section, we tackle the solution to the optimization problem (3), with $\bar{D}_{\mu}$ given by (II-C). However, (3) cannot be directly solved using dynamic programming techniques, due to the presence of the denominator in (II-C), which depends on the policy selected $\mu$, hence it affects the optimization. The next lemma demonstrates that the denominator of (II-C) can be expressed as a positive constant, independent from policy $\mu$ and only dependent on system parameters. In doing so, the optimization of $\mu$ only needs to focus on the minimization of $\sum_{s\in\mathcal{S}}\Pi_{\mu}(s)\Delta_{s}(\mu(s))$, so that (3) can be cast as an average cost per stage problem, solvable with standard dynamic programming techniques.

Lemma 1**.**

Let $\pi_{\mathrm{wait}}$ and $\pi_{\mathrm{comm}}$ be the steady-state probabilities that the UAV is in the waiting and communication phases, $\pi_{\mathrm{comm}}{=}\sum_{s\in\mathcal{S}_{\mathrm{comm}}}\!\!\!\!\!\Pi_{\mu}(s)$ and $\pi_{\mathrm{wait}}{=}1{-}\pi_{\mathrm{comm}}$. We have that

[TABLE]

Proof.

Let $p_{ww}$, $p_{wc}$, $p_{cw}$, and $p_{cc}$ be the probabilities of a state request status, $r\in\mathcal{R}=\{0,1,2\}$, transitioning in the SMDP as $0{\rightarrow}0$, $0{\rightarrow}\{1,2\}$, $\{1,2\}{\rightarrow}0$, and $\{1,2\}{\rightarrow}\{1,2\}$, respectively. Then, $p_{ww}=e^{-\lambda\Delta_{0}}$ (if no request is received, the SMDP remains in the waiting state), $p_{wc}=1-p_{ww}$, $p_{cw}=1$, and $p_{cc}=0$ (if the SMDP is in the communication state, the next state of the SMDP will be a waiting state, see (9)). Therefore, the steady-state probabilities of being in the waiting and communication states, $\pi_{\mathrm{wait}}$ and $\pi_{\mathrm{comm}}$, satisfy

[TABLE]

whose solution is given as in the statement of the lemma. ∎

When we refer to the denominator of (II-C), it is evident that it is equal to the steady-state probability that the UAV is in a communication state while following policy $\mu$, $\pi_{\mathrm{comm}}$. However, with the result of Lemma 1, $\pi_{\mathrm{comm}}$ is simply a positive constant determined by system parameters, yielding

[TABLE]

which we now aim to minimize with respect to policy $\mu$.

As the problem stands now, the communication phase selects an action from $\mathcal{T}_{r}(i)$, which is a set containing an uncountable number of trajectories. By exploiting the two-scale structure of the problem outlined earlier, we now demonstrate that only a finite set of trajectories from $\mathcal{T}_{r}(i)$ are eligible to be optimal, for each state $(i,r)\in\mathcal{S}_{\mathrm{comm}}$, hence making the problem a finite state and action SMDP.

III-A Decomposition of Policy $\mu$

Note from (9) that the transition probability from a communication state $s_{n}{=}(i,r)$ under action $(j,q(\cdot))$ is only affected by the selection of $j$ and not the particular trajectory $q(\cdot){\in}\mathcal{T}_{r}(i\rightarrow j)$ that leads from $q_{i}$ to $q_{j}$ during the communication phase. It follows that the steady-state probability $\Pi_{\mu}$ under $\mu(i,r){=}(j,q(\cdot))$ is only affected by the selection of $j$ and not the specific trajectory within $\mathcal{T}_{r}(i\rightarrow j)$.

By establishing this property, we decompose the policy $\mu$ into the waiting policy $\theta(i){\in}\{-1,0,1\}$, which defines the optimal action in state $(i,0){\in}\mathcal{S}_{\mathrm{wait}}$ of the waiting phase; the end position policy $J(i,r)$, which selects the end position $q_{j}$ with $j{=}J(i,r)$ to be reached at the end of the communication phase; and the trajectory policy $\rho(i,r,j)$ which, given $j{=}J(i,r)$, selects a trajectory $q(\cdot){=}\rho(i,r,j)$ from $\mathcal{T}_{r}(i\to j)$. Owing to the independence of $\Pi_{\mu}$ on the trajectory policy $\rho$, the delay minimization problem can then be rewritten as

[TABLE]

Letting

[TABLE]

we can finally write

[TABLE]

Note that $\Delta_{r}^{*}(i,j)$ yields the trajectory that greedily minimizes the communication delay when starting from state $(i,r)$, ending in position $q_{j}$ while serving GNr. This result proves that, for any communication state $(i,r)$, there exist only $2N+1$ trajectories that are eligible to be optimal, one for each possible ending position $q_{j}\in\mathcal{Q}$. Hence, the problem is finally reduced to that of finding the optimal waiting policy $\theta$ and end position policy $J$, which can be solved efficiently via dynamic programming (Algorithm 1). In the next section, we provide a closed form expression of $\Delta_{r}^{*}(i,j)$.

III-B Closed-form Delay Minimizing Trajectory

With the independence of the steady-state probabilities from $\rho$, we can proceed to solve (14) and then provide the dynamic programming algorithm to solve for $\theta^{*}$ and $J^{*}$ in (15). By definition of $\mathcal{T}_{r}(i{\rightarrow}j)$ in (II-C), we can rewrite $\Delta_{r}^{*}(i,j)$ as

[TABLE]

The minimizer $q^{*}$ is the trajectory that the UAV should follow when receiving a request from GNr starting in position $q_{i}$ and ending in position $q_{j}$, selected by the end position policy $J$.

In defining the optimal trajectory, the following definitions will be useful. Let $\tau_{p_{1},p_{2}}{\triangleq}\frac{|p_{2}-p_{1}|}{V}$ be the time needed to fly at maximum speed from $p_{1}$ to $p_{2}{\in}[-a,a]$. Along this straight trajectory, let

[TABLE]

be the amount of bits transmitted to serve GNr.

Clearly, $\ell_{p_{1},p_{1}}^{(r)}{=}0$ ($\tau_{p_{1},p_{1}}{=}0$), $\ell_{p_{1},p_{2}}^{(r)}{=}\ell_{p_{2},p_{1}}^{(r)}$ ($\tau_{p_{1},p_{2}}{=}\tau_{p_{2},p_{1}}$), and $\ell_{p_{1},p_{2}}^{(1)}{=}\ell_{-p_{1},-p_{2}}^{(2)}$ ($\tau_{p_{1},p_{2}}{=}\tau_{-p_{1},-p_{2}}$). The integral $\ell_{p_{1},p_{2}}^{(r)}$ can be determined in closed form and is found in [2], for example. We also define the trajectory $\upsilon\{p_{1}{\to}(p_{2},\delta){\to}p_{3}\}(\tau),\ \tau\in[0,\delta{+}\tau_{p_{1},p_{2}}{+}\tau_{p_{2},p_{3}}]$, as the one in which the UAV starts at position $p_{1}$, flies at maximum speed to $p_{2}$, hovers at $p_{2}$ for $\delta$ amount of time, and finally flies at maximum speed from $p_{2}$ to $p_{3}$. Mathematically,

[TABLE]

The traffic delivered to GNr when following this trajectory is $\ell_{p_{1},p_{2}}^{(r)}\!\!{+}\delta R_{r}(p_{2}){+}\ell_{p_{2},p_{3}}^{(r)}$, with delay $\tau_{p_{1},p_{2}}\!\!{+}\delta{+}\tau_{p_{2},p_{3}}$. With these definitions, we are now ready to state the main result.

Theorem 1**.**

Let $q^{*}(\cdot){\in}\mathcal{T}_{r}(i\rightarrow j)$ be the trajectory that minimizes the communication delay $\Delta_{r}^{*}(i,j)$. If $\ell_{q_{i},q_{j}}^{(r)}{\geq}L$, then

[TABLE]

i.e., the UAV flies at maximum speed from $q_{i}$ to $q_{j}$ without interruption; otherwise, if $\ell_{q_{i},x_{r}}^{(r)}+\ell_{x_{r},q_{j}}^{(r)}\leq L$, then

[TABLE]

where

[TABLE]

i.e., the UAV flies at maximum speed from $q_{i}$ to $x_{r}$, hovers over $x_{r}$ for $\delta^{*}$ amount of time, and then flies to $q_{j}$; finally, if $\ell_{q_{i},x_{r}}^{(r)}+\ell_{x_{r},q_{j}}^{(r)}>L$, but $\ell_{q_{i},q_{j}}^{(r)}<L$, then

[TABLE]

where $p^{*}$ is the unique solution in $[x_{r},\min\{q_{i},q_{j}\}]$ (if $r{=}1$) or $[\max\{q_{i},q_{j}\},x_{r}]$ (if $r{=}2$) of $\ell_{q_{i},p^{*}}^{(r)}{+}\ell_{p^{*},q_{j}}^{(r)}{=}L$; i.e., the UAV flies at maximum speed towards $x_{r}$ to the farthest point $p^{*}$ and then back to $q_{j}$, with $p^{*}$ uniquely defined in such a way as to transmit exactly the data payload upon reaching $q_{j}$.

Proof.

Due to space limitations, we provide an outline of the proof. Assume $r{=}2$ (a similar argument applies to $r{=}1$ by symmetry).

1. for any trajectory $q(\cdot)\in\mathcal{T}_{2}(i\to j)$ of duration $\Delta$, one can construct another trajectory $\tilde{q}(\cdot)\in\mathcal{T}_{2}(i\to j)$ of same duration $\Delta$, and such that $|q(t)-x_{r}|\geq|\tilde{q}(t)-x_{r}|,\ \forall t\in[0,\Delta]$; such trajectory is obtained by flying at maximum speed towards GN2, possibly hovering on top of GN2 for $\delta$ amount of time (if time allows), and then returning to $q_{j}$, yielding $\tilde{q}(\cdot){=}\upsilon\{q_{i}{\to}(p^{*},\delta^{*}){\to}q_{j}\}(\cdot)$, for a proper choice of $p^{*}$ and $\delta^{*}$ such that $\tau_{q_{i},p^{*}}{+}\tau_{p^{*},q_{j}}{+}\delta{=}\Delta$;
2. note that the UAV is always closer to GN2 under $\tilde{q}(\cdot)$ than it is under $q(\cdot)$, hence it delivers a larger data payload than $q(\cdot)$ while incurring the same delay; therefore, $q(\cdot)$ is suboptimal;
3. $\tilde{q}(\cdot)$ can be further improved by minimizing the delay (by optimizing $(p^{*},\delta^{*})$), yielding the three cases provided in the statement of the theorem.∎

III-C Multi-chain Policy Iteration (PI) Algorithm

We opt to use a multi-chain PI algorithm to solve (15), as there exist some policies whose induced Markov chain structures are multi-chain. For example, if the waiting policy is $\theta(i)=0$, and the end position policy is $J(i,r)=i$, then the induced Markov chain has $2N+1$ recurrent classes (hence multi-chain). To accommodate this structure, the pseudocode that follows is based upon the multi-chain PI methods of [11] and succinctly describes how to solve for $\mu^{*}$.

In Algorithm 1, we use a vector notation for $\mathbf{\bar{D}}_{k}$ and $\mathbf{h}_{k}$, which denote the average delay and relative value for all states, respectively, following the $k$th policy iterate $\mu_{(k)}$. Likewise, $\mathbf{c}_{\mu}$ is the vector notation for the delay cost function under policy $\mu$, supplemented by the optimal minimized communication delays described by (14) and (III-B), and $\mathbf{P}_{\mu}$ is the transition matrix under policy $\mu$.

IV Numerical Results

We use the following system parameters, unless specified otherwise: number of states $2N{+}1{=}101$; channel bandwidth $B{=}1\mathrm{MHz}$; $1$-meter reference SNR $\gamma_{\mathrm{dB}}{=}40\mathrm{dB}$; UAV height $H{=}100\mathrm{m}$; GN locations $x_{1}{=}-400\mathrm{m}$, $x_{2}{=}400\mathrm{m}$; UAV speed $V{=}20\mathrm{m/s}$; and request arrival rate $\lambda{=}0.4$ requests/second.

We vary the data payload $L$ across a range of values and find numerically that, regardless, the optimal policy in the waiting phase optimized with Algorithm 1 is

[TABLE]

In other words, in the waiting phase it is optimal for the UAV to move towards the geometric center of the two GNs along the line segment connecting the two. Intuitively, the UAV can more readily service a request that is originated equally likely from GN1 or GN2, if it is located in the geometric center when the request arrives, since the UAV is equally distant from both GNs, and can thus serve them equally well.

In Fig. 2, we plot the optimal end position policy $J^{*}(i,2)$ for different data payload values.333We omit the figure for states $(i,1)\in\mathcal{S}_{\mathrm{comm}}$, due to the inherent symmetry of the problem. Specifically, if the optimal end point $J^{*}(i,2)=j$ is observed, then $J^{*}(-i,1)=-j$ is also observed. We note that, for large data payload values $L$, the optimal end position in the communication phase becomes independent of the initial position $i$ (in this case, $J^{*}(i,2){\approx}336\mathrm{m}$, irrespective of $i$ for $L{\gg}1$). In fact, for large data payload $L$, the UAV hovers over the receiver for a significant amount of time during the communication phase (case $\ell_{q_{i},x_{r}}^{(r)}{+}\ell_{x_{r},q_{j}}^{(r)}{\leq}L$ in Theorem 1), hence the final part of the trajectory from $x_{r}$ to the selected end position $q_{j}$ becomes irrespective of the actual data payload value. However, $J^{*}(i,2)$ does depend on other system parameters, such as the request rate $\lambda$ and UAV height $H$, as seen in Fig. 3. Interestingly, as the request rate increases (the inter-arrival request time $1/\lambda$ decreases) the end position is closer to the geometric center (i.e., farther away from the receiver); this is because requests arrive more often, hence it is desirable for the UAV to terminate the communication phase closer to the center, in order to more readily serve future requests.

Next, we illustrate how the optimal expected average delay $\bar{D}_{\mu}^{*}$, across the same set of data payload values, fares against a heuristic policy which operates as follows: in the waiting phase, hover in the current position; in the data communication phase, greedily minimize the delay by flying at maximum speed towards the receiver until completion. The comparison between the optimal policy $\mu^{*}$ and the heuristic policy is shown for the span of data payload values in Fig. 4. Note that the slope of the line for both the optimal and heuristic policies saturates to $[B\log_{2}(1{+}\gamma/H^{2})]^{-1}$. In fact, when $L{\gg}1$, the UAV spends most of the communication time hovering above the receiver (case $\ell_{q_{i},x_{r}}^{(r)}{+}\ell_{x_{r},q_{j}}^{(r)}{\leq}L$ in Theorem 1), hence $\Delta_{r}^{*}(i,j){\approx}\frac{L}{R_{r}(x_{r})}$ in (15), yielding

[TABLE]

Overall, the heuristic scheme performs worse, roughly by $2$ seconds for large $L$. In fact, when hovering during the waiting phase instead of moving towards the center, the UAV incurs a larger delay to serve a request generated by the more distant GN, due to the longer distance that needs to be covered.

V Conclusions

In this paper, we studied the trajectory optimization problem of one UAV servicing random downlink transmission requests by two GNs, to minimize the expected communication delay. We formulated the problem as an SMDP, and exploited the structure of the problem to simplify the trajectory design in the communication phase. We showed that the problem exhibits an interesting two-scale structure in the optimal trajectory design, and can be solved efficiently via dynamic programming. Numerical evaluations demonstrate consistent improvements in the delay performance over a sensible heuristic, for a variety of data payload values.