Delay-Optimal Probabilistic Scheduling with Arbitrary Arrival and Adaptive Transmission

TL;DR

This paper develops a delay-power tradeoff framework for adaptive transmission scheduling with arbitrary arrivals, using Markov decision processes and reinforcement learning to find optimal policies efficiently.

Contribution

It introduces a novel CMDP-based model for delay-power tradeoff, characterizes the optimal policy as threshold-based, and proposes efficient algorithms including reinforcement learning for unknown arrival distributions.

Findings

Optimal tradeoff curve is decreasing, convex, and piecewise linear.

Optimal policy is threshold-based.

Reinforcement learning effectively finds policies with unknown arrival distributions.

Abstract

In this paper, we aim to obtain the optimal delay-power tradeoff and the corresponding optimal scheduling policy for an arbitrary i.i.d. arrival process and adaptive transmissions. The number of backlogged packets at the transmitter is known to a scheduler, who has to determine how many backlogged packets to transmit during each time slot. The power consumption is assumed to be convex in transmission rates. Hence, if the scheduler transmits faster, the delay will be reduced but with higher power consumption. To obtain the optimal delay-power tradeoff and the corresponding optimal policy, we model the problem as a Constrained Markov Decision Process (CMDP), where we minimize the average delay given an average power constraint. By steady-state analysis and Lagrangian relaxation, we can show that the optimal tradeoff curve is decreasing, convex, and piecewise linear, and the optimal policy is threshold-based. Based on the revealed properties of the optimal policy, we develop an algorithm to efficiently obtain the optimal tradeoff curve and the optimal policy with full information of the system. The complexity of our proposed algorithm is much lower than a general algorithm based on Linear Programming. However, usually the distribution of the arrival process is unknown to the scheduler, therefore we proposed a reinforcement learning algorithm to efficiently obtain the optimal policy under this circumstance. We also analyse in details about how the system parameters affect the optimal policy and the system performance. In the final, we use simulations to validate the derived results and the proposed algorithms.

Equations

$$q[n+1]=q[n]-s[n]+a[n].$$

$$\text{Pr}\{q[n+1]=j|q[n]=q,s[n]=s\}$$

$$=$$

$$f_{q,s}=\text{Pr}\{s[n]=s|q[n]=q\}.$$

$$f_{q,s}=0\text{ if }q-s<0\text{ or }q-s>Q-A.$$

$$\underset{\mathbf{F}\in \mathcal{F}}{\min }$$

$$P_{\mathbf{F}}\le P_{\text{th}}.$$

$${\lambda }_{i,j}=\sum _{s=\max \{0,i+A-Q,i-j\}}^{\min \{S,i,i-j+A\}}{\alpha }_{j-i+s}{f}_{i,s}.$$

$${\mathbf{H}}_{\mathbf{F}}{\mathbf{\pi }}_{\mathbf{F}}=\mathbf{c}.$$

$$P_{\mathbf{F}}=\sum _{q=0}^Q{\pi }_{\mathbf{F}}(q)\sum _{s=0}^S{P}_s{f}_{q,s}={\mathbf{p}}_{\mathbf{F}}^T{\mathbf{\pi }}_{\mathbf{F}}.$$

$$D_{\mathbf{F}}=\frac{1}{E_a}\sum _{q=0}^{Q}q{\pi }_{\mathbf{F}}(q)=\frac{1}{E_a}{\mathbf{d}}^T{\mathbf{\pi }}_{\mathbf{F}}.$$

$$\underset{\mathbf{F}\in \mathcal{F}}{\min }{D}_{\mathbf{F}}+\mu P_{\mathbf{F}}-\mu P_{\text{th}}.$$

$$\{\begin{array}{ll}{f}_{q,s}=1& q_{\mathbf{F}}(s-1)<q\le q_{\mathbf{F}}(s),s=0,\cdots ,S\\ f_{q,s}=0& \text{otherwise}\end{array}$$

$$\displaystyle\left\{\begin{array}[]{@{}ll}f_{q,s}=1&q_{\boldsymbol{F}}(s-1)<q\leq q_{\boldsymbol{F}}(s),s\neq s^{*}\\ f_{q,s^{*}}=1&q_{\boldsymbol{F}}(s^{*}-1)<q<q_{\boldsymbol{F}}(s^{*})\\ \lx@intercol f_{q_{\boldsymbol{F}}(s^{*}),s^{*}}+f_{q_{\boldsymbol{F}}(s^{*}),s^{*}+1}=1\hfil\lx@intercol\\ f_{q,s}=0&\text{otherwise}\end{array}\right.$$

$$\min$$

$$\sum _{q=0}^Q\sum _{s=0}^S{P}_s{x}_{q,s}\le P_{\text{th}}$$

$$\sum _{l=\max \{0,q-A\}}^{q-1}\sum _{a=0}^A\sum _{s=0}^{l+a-q}{\alpha }_a{x}_{l,s}$$

$$=\sum _{r=q}^{\min \{q+S-1,Q\}}\sum _{a=0}^A\sum _{s=r+a-q+1}^S{\alpha }_a{x}_{r,s}q=1,\cdots ,Q$$

$$\sum _{q=0}^Q\sum _{s=0}^S{x}_{q,s}=1$$

$$x_{q,s}=0\forall q-s<0\text{ or }q-s>Q-A$$

$$x_{q,s}\ge 0\forall 0\le q-s\le Q-A.$$

$${\mathbf{H}}_{\mathbf{F}}^{-1}={\left[{\mathbf{h}}_0^T,{\mathbf{h}}_1^T,\cdots ,{\mathbf{h}}_Q^T\right]}^T.$$

$$\displaystyle(\boldsymbol{H}_{\boldsymbol{F}}^{-1}\Delta\boldsymbol{H})\boldsymbol{H}_{\boldsymbol{F}}^{-1}=\left[\begin{array}[]{c}(\boldsymbol{h}_{0}^{T}\boldsymbol{\delta}_{q})\boldsymbol{h}_{q}^{T}\\ (\boldsymbol{h}_{1}^{T}\boldsymbol{\delta}_{q})\boldsymbol{h}_{q}^{T}\\ \vdots\\ (\boldsymbol{h}_{Q}^{T}\boldsymbol{\delta}_{q})\boldsymbol{h}_{q}^{T}\end{array}\right].$$

$$({\mathbf{H}}_{\mathbf{F}}^{-1}\Delta \mathbf{H}{)}^i{\mathbf{H}}_{\mathbf{F}}^{-1}$$

$$=$$

$$\Delta {\mathbf{p}}^T{\mathbf{H}}_{\mathbf{F}}^{-1}({\mathbf{H}}_{\mathbf{F}}^{-1}\Delta \mathbf{H}{)}^{i-1}$$

$$=$$

$$({\mathbf{H}}_{\mathbf{F}}+ϵ\Delta \mathbf{H}{)}^{-1}$$

$$=$$

$$\frac{P_{{\mathbf{F}}^{\prime \prime }}-P_{\mathbf{F}}}{P_{{\mathbf{F}}^{\prime }}-P_{\mathbf{F}}}$$

$$=$$

$$\frac{D_{{\mathbf{F}}^{\prime \prime }}-D_{\mathbf{F}}}{D_{{\mathbf{F}}^{\prime }}-D_{\mathbf{F}}}$$

$$=$$

Full text

I Introduction

In this paper, we study an important problem of how to schedule the number of packets to transmit over a link taking into account both the delay and the power cost. This is an important problem because delay is a vital metric for many emerging applications (e.g., instant messenger, social network service, streaming media, and so on), and power consumption is critical to battery life of various mobile devices. In other words, we are studying the tradeoff between the timeliness and greenness of the communication service.

Such a delay-power scheduling problem can be formulated using a Markov Decision Process (MDP). The authors in [1] were among the earliest who studied this type of scheduling problem. Specifically, they considered a two-state channel and finite time horizon. The dual problem was solved based on results derived by Dynamic Programming and induction. Follow-up papers [2, 3, 4, 5] extended this study in various directions. The optimal delay-power tradeoff curve is proven to be nonincreasing and convex in [2]. The existence of stationary optimal policy and the structure of the optimal policy are further investigated in [3]. Different types of power/rate control policies are studied in [4]. In [5], the asymptotic small-delay regime is investigated. In [6], a piecewise linear delay-power tradeoff curve was obtained along with an approximate closed form expression.

If one can show monotonicity or a threshold type of structure to the optimal policy for MDPs, it helps to substantially reduce the computation complexity in finding the optimal policy. Indeed, the optimal scheduling policies are shown to be threshold-based or monotone in [1, 3, 5, 7, 8, 9, 10], proven by studying the convexity, superadditivity / subadditivity, or supermodularity / submodularity of expected cost functions by induction using dynamic programming. However, most of these results are limited to the unconstrained Lagrangian Relaxation problem. In [3, 10], some properties of the optimal policy for the constrained problem are described based on the results for the unconstrained problem. Detailed analysis on the optimal policy for the constrained problem is conducted in [8, 9]. In [8], properties such as unichain policies and multimodularity of costs are assumed to be true so that monotone optimal policies can be proven. In [9], the transmission action is either 1 or 0, i.e. to transmit or not. In order to obtain the detailed structure of the solution to the constrained problem, we believe that the analysis of the Lagrangian relaxation problem and the analysis of the structure of the delay-power tradeoff curve should be combined together.

In [11], we study the optimal delay-power tradeoff problem. In particular, we minimize the average delay given an average power constraint, considering Bernoulli arrivals and adaptive transmissions. Some technical details are given in [12], where we proved that the optimal tradeoff curve is convex and piecewise linear, and the optimal policies are threshold-based, by Constrained Markov Decision Process formulation and steady-state analysis. In this paper, we substantially generalize the Bernoulli arrival process to an arbitrary i.i.d. distribution. We show that the optimal policies for this generalized model are still threshold-based. Furthermore, we develop an efficient algorithm to find the optimal policy and the optimal delay-power tradeoff curve.

The remainder of this paper is organized as follows. The system model and the constrained problem are introduced in Section II. We show that the optimal policy is threshold-based in Section III by using steady-state analysis and Lagrangian relaxation. Based on theoretical results, we propose an efficient algorithm in Section IV to obtain the optimal tradeoff curve and the corresponding policies. In Section V, theoretical results and the proposed algorithm are verified by simulations. Section VI concludes the paper.

II System Model

The system model is shown in Figure 1. We assume there are $a[n]$ data packet(s) arriving at the end of the $n$th timeslot. The number $a[n]$ is i.i.d. for different values of $n$ and its distribution is given by $\text{Pr}\{a[n]=a\}=\alpha_{a}$, where $\alpha_{a}\geq 0$, $a\in\{0,1,\cdots,A\}$, and $\sum_{a=0}^{A}\alpha_{a}=1$. Therefore the expected number of packets arrived in each timeslot $n$ is given by $E_{a}=\sum_{a=0}^{A}a\alpha_{a}$.

Let $s[n]$ denote the number of data packets transmitted in timeslot $n$. Assume that at most $S$ packets can be transmitted in each timeslot because of the constraints of the transmitter, and $S\geq A$. Let $\tau[n]$ denote the transmission power consumed in timeslot $n$. Assume transmitting $s$ packet(s) will cost power $P_{s}$, where $s\in\{0,1,\cdots,S\}$, therefore $\tau[n]=P_{s[n]}$. Transmitting [math] packet will cost no power, hence $P_{0}=0$. In typical communications, the power efficiency decreases as the transmission rate increases, hence we assume that $P_{s}$ is convex in $s$. Detailed explanations can be found in the Introduction section in [12]. The convexity of the power consumption function will be utilized in Theorem 2 to prove that the optimal policy for the unconstrained problem is threshold-based.

Backlog packets are stored in a buffer with size $Q$. Let $q[n]\in\{0,1,\cdots,Q\}$ denote the queue length at the beginning of timeslot $n$. Since data arrive at the end of the timeslot, in order to avoid buffer overflow (i.e. $q[n]>Q$) and underflow (i.e. $q[n]<0$), we should have $0\leq q[n]-s[n]\leq Q-A$. Therefore the dynamics of the buffer is given as

[TABLE]

In timeslot $n$, we can decide how many packets to be transmitted based on the buffer state $q[n]$. It can be seen that this is a Markov Decision Process (MDP), where the queue length $q[n]$ is the state of the MDP, and the number of packets transmitted in each timeslot $s[n]$ is the action we take in each timeslot $n$. The probability distribution of the next state $q[n+1]$ is given by

[TABLE]

We minimize the average queueing delay given an average power constraint, which makes it a Constrained Markov Decision Process (CMDP). For an infinite-horizon CMDP with stationary parameters, according to [13, Theorem 11.3], stationary policies are complete, which means stationary policies can achieve the optimal performance. Therefore we only need to consider stationary policies in this problem. Let $f_{q,s}$ denote the probability to transmit $s$ packet(s) when $q[n]=q$, i.e.,

[TABLE]

Then we have $\sum_{s=0}^{S}f_{q,s}=1$ for $q=0,\cdots,Q$. Since we guarantee that the transmission strategy will avoid overflow or underflow, we set

[TABLE]

Let $\boldsymbol{F}$ denote a $(Q+1)\times(S+1)$ matrix whose element in the $(q+1)$th row and the $(s+1)$th column is $f_{q,s}$. Therefore matrix $\boldsymbol{F}$ can represent a stationary transmission policy. Let $P_{\boldsymbol{F}}$ and $D_{\boldsymbol{F}}$ denote the average power consumption and the average queueing delay under policy $\boldsymbol{F}$. Let $\mathcal{F}$ denote the set of all feasible stationary policies that guarantee no queue overflow or underflow. Let $\mathcal{F}_{D}$ denote the set of all stationary and deterministic policies which can guarantee no overflow or underflow. Thus to obtain the optimal tradeoff curve, we can minimize the average delay given an average power constraint $P_{\text{th}}$ shown as

[TABLE]

From another perspective, policy $\boldsymbol{F}$ will determine a point $Z_{\boldsymbol{F}}=(P_{\boldsymbol{F}},D_{\boldsymbol{F}})$ in the delay-power plane. Define $\mathcal{R}=\{Z_{\boldsymbol{F}}|\boldsymbol{F}\in\mathcal{F}\}$ as the set of all feasible points in the delay-power plane. Intuitively, since the power consumption for each data packet increases if we want to transmit faster, there is a tradeoff between the average queueing delay and the average power consumption. Thus the optimal delay-power tradeoff curve can be presented as $\mathcal{L}=\{(P,D)\in\mathcal{R}|\forall(P^{\prime},D^{\prime})\in\mathcal{R},\text{ either }P^{\prime}\geq P\text{ or }D^{\prime}\geq D\}$.

If we fix a stationary policy for a Markov Decision Process, the Markov Decision Process will degenerate to a Markov Reward Process (MRP). Let $\lambda_{i,j}$ denote the transition probability from state $i$ to state $j$. According to the system model, because of the constraints of transmission and arrival processes, the state transition probability can be derived as

[TABLE]

An example of the transition diagram is shown in Figure 2, where $\lambda_{i,i}$ for $i=0,\cdots,Q$ are omitted to keep the diagram legible.

The Markov chain could have more than one closed communication classes under certain transmission policies. Under this circumstance, the limiting probability distribution and the average cost are dependent on the initial state and the sample paths. In Appendix A, it is proven that we only need to consider the cases where the Markov chain has only one closed communication class, which is called a unichain. Becausae of this key result, we focus only on the unichain cases in the following.

III Optimal Threshold-Based Policy for the Constrained Markov Decision Process

In this section, we will demonstrate that the optimal policy for the Constrained MDP problem is threshold-based. In other words, for an optimal policy, more data will be transmitted if the queue is longer. We give the rigorous definition of a stationary threshold-based policy $\boldsymbol{F}$ that, there exist $(S+1)$ thresholds $0\leq q_{\boldsymbol{F}}(0)\leq q_{\boldsymbol{F}}(1)\leq\cdots\leq q_{\boldsymbol{F}}(S)\leq Q$, such that $f_{q,s}>0$ only when $q_{\boldsymbol{F}}(s-1)\leq q\leq q_{\boldsymbol{F}}(s)$ (set $q_{\boldsymbol{F}}(-1)=-1$ for simplicity of notation). According to this definition, under policy $\boldsymbol{F}$, when the queue state is larger than threshold $q_{\boldsymbol{F}}(s-1)$ and smaller than $q_{\boldsymbol{F}}(s)$, it transmits $s$ packet(s). When the queue state is equal to threshold $q_{\boldsymbol{F}}(s)$, it transmits $s$ or $(s+1)$ packet(s). Note that under this definition, probabilistic policies can also be threshold-based.

In the following, we will first conduct the steady-state analysis of the Markov process, based on which we can show the properties of the feasible delay-power region and the optimal delay-power tradeoff, and then by proving that the Lagrangian relaxation problem has a deterministic and threshold-based optimal policy, we can finally show that the optimal policy for the constrained problem is threshold-based.

III-A Steady State Analysis

Since we can focus on unichain cases, which contain a single recurrent class plus possibly some transient states, the steady-state probability distribution exists for the Markov process. Let $\pi_{\boldsymbol{F}}(q)$ denote the steady-state probability for state $q$ when applying policy $\boldsymbol{F}$. Set $\boldsymbol{\pi}_{\boldsymbol{F}}=[\pi_{\boldsymbol{F}}(0),\cdots,\pi_{\boldsymbol{F}}(Q)]^{T}$. Define $\boldsymbol{\Lambda}_{\boldsymbol{F}}$ as a $(Q+1)\times(Q+1)$ matrix whose element in the $(i+1)$th column and the $(j+1)$th row is $\lambda_{i,j}$, which is determined by policy $\boldsymbol{F}$. Set $\boldsymbol{I}$ as the identity matrix. Define $\boldsymbol{1}=[1,\cdots,1]^{T}$, and $\boldsymbol{0}=[0,\cdots,0]^{T}$. Set $\boldsymbol{G}_{\boldsymbol{F}}=\boldsymbol{\Lambda}_{\boldsymbol{F}}-\boldsymbol{I}$. Set \boldsymbol{H}_{\boldsymbol{F}}=\left[\begin{array}[]{c}\boldsymbol{1}^{T}\\ \boldsymbol{G}_{\boldsymbol{F}}(0:(Q-1),:)\end{array}\right] and \boldsymbol{c}=\left[\begin{array}[]{c}1\\ \boldsymbol{0}\end{array}\right].

According to the definition of the steady-state distribution, we have $\boldsymbol{G}_{\boldsymbol{F}}\boldsymbol{\pi}_{\boldsymbol{F}}=\boldsymbol{0}$ and $\boldsymbol{1}^{T}\boldsymbol{\pi}_{\boldsymbol{F}}=1$. For a unichain, the rank of $\boldsymbol{G}_{\boldsymbol{F}}$ is $Q$. Therefore, we have $\boldsymbol{H}_{\boldsymbol{F}}$ is invertible and

[TABLE]

For state $q$, transmitting $s$ packet(s) will cost $P_{s}$ with probability $f_{q,s}$. Define $\boldsymbol{p}_{\boldsymbol{F}}=[\sum_{s=0}^{S}P_{s}f_{0,s},\cdots,\sum_{s=0}^{S}P_{s}f_{Q,s}]^{T}$, which is a function of $\boldsymbol{F}$. The average power consumption $P_{\boldsymbol{F}}$ can be expressed as

[TABLE]

Similarly, define $\boldsymbol{d}=[0,1,\cdots,Q]^{T}$. According to Little’s Law, the average delay $D_{\boldsymbol{F}}$ under policy $\boldsymbol{F}$ is

[TABLE]

The following theorem describes the structure of the feasible delay-power region and the optimal delay-power tradeoff curve.

Theorem 1**.**

The set of all feasible points in the delay-power plane, $\mathcal{R}$, and the optimal delay-power tradeoff curve $\mathcal{L}$, satisfy that

The set $\mathcal{R}$ is a convex polygon. 2. 2.

The curve $\mathcal{L}$ is piecewise linear, decreasing, and convex. 3. 3.

Vertices of $\mathcal{R}$ and $\mathcal{L}$ are all obtained by deterministic scheduling policies. 4. 4.

The policies corresponding to adjacent vertices of $\mathcal{R}$ and $\mathcal{L}$ take different actions in only one state.

Proof:

See Appendix B. ∎

III-B Optimal Deterministic Threshold-Based Policy for the Lagrangian Relaxation Problem

In (5), we formulate the optimization problem as a Constrained MDP, which is difficult to solve in general. Let $\mu\geq 0$ denote the Lagrange multiplier. Consider the Lagrangian relaxation of (5)

[TABLE]

In (10), the term $-\mu P_{\text{th}}$ is constant. Therefore, the Lagrangian relaxation problem is minimizing the weighted average cost $D_{\boldsymbol{F}}+\mu P_{\boldsymbol{F}}$, which becomes an unconstrained infinite-horizon Markov Decision Process with an average cost criterion. It is proven in [14, Theorem 9.1.8] that, there exists an optimal stationary deterministic policy. Moreover, the optimal policy for the relaxation problem has the following property.

Theorem 2**.**

An optimal policy $\boldsymbol{F}$ for the unconstrained Markov Decision Process is threshold-based. That is to say, there exists $(S+1)$ thresholds $q_{\boldsymbol{F}}(0)\leq q_{\boldsymbol{F}}(1)\leq\cdots\leq q_{\boldsymbol{F}}(S)$, such that

[TABLE]

where $q_{\boldsymbol{F}}(-1)=-1$.

Proof:

See Appendix C. ∎

III-C Optimal Threshold-Based Policy for the Constrained Problem

From another perspective, $D_{\boldsymbol{F}}+\mu P_{\boldsymbol{F}}=\langle(\mu,1),(P_{\boldsymbol{F}},D_{\boldsymbol{F}})\rangle$ can be seen as the inner product of vector $(\mu,1)$ and $Z_{\boldsymbol{F}}$. Since $\mathcal{L}$ is piecewise linear, decreasing and convex, the corresponding $Z_{\boldsymbol{F}}$ minimizing the inner product will be obtained by the vertices of $\mathcal{L}$, as can be observed in Figure 3. Since the conclusion in Theorem 2 holds for any $\mu$, the vertices of the optimal tradeoff curve can all be obtained by optimal policies for the Lagrangian relaxation problem, which are deterministic and threshold-based. Moreover, from Theorem 1, the adjacent vertices of $\mathcal{L}$ are obtained by policies which take different actions in only one state. Therefore, we can have the following theorem.

Theorem 3**.**

Given an average power constraint, the scheduling policy $\boldsymbol{F}$ to minimize the average delay takes the following form: there exists $(S+1)$ thresholds $q_{\boldsymbol{F}}(0)\leq q_{\boldsymbol{F}}(1)\leq\cdots\leq q_{\boldsymbol{F}}(S)$, one of which we name $q_{\boldsymbol{F}}(s^{*})$, such that

[TABLE]

where $q_{\boldsymbol{F}}(-1)=-1$.

Proof:

Since the optimal tradeoff curve is piecewise linear, assume $Z_{\boldsymbol{F}}$ is on the line segment between vertices $Z_{\boldsymbol{F}^{\prime}}$ and $Z_{\boldsymbol{F}^{\prime\prime}}$. According to Theorem 2, the form of optimal policies $\boldsymbol{F}^{\prime}$ and $\boldsymbol{F}^{\prime\prime}$, which are corresponding to vertices of the optimal tradeoff curve, satisfies (11). Moreover, according to Theorem 1, the policies corresponding to adjacent vertices of $\mathcal{L}$ take different actions in only one state. Define the thresholds for $\boldsymbol{F}^{\prime}$ as $q_{\boldsymbol{F}^{\prime}}(0),q_{\boldsymbol{F}^{\prime}}(1),\cdots,q_{\boldsymbol{F}^{\prime}}(s^{*}),\cdots,q_{\boldsymbol{F}^{\prime}}(S)$, then the thresholds for $\boldsymbol{F}^{\prime}$ can be expressed as $q_{\boldsymbol{F}^{\prime}}(0),q_{\boldsymbol{F}^{\prime}}(1),\cdots,q_{\boldsymbol{F}^{\prime}}(s^{*})-1,\cdots,q_{\boldsymbol{F}^{\prime}}(S)$, where the two policies take different actions only in state $q_{\boldsymbol{F}^{\prime}}(s^{*})$. Since $Z_{\boldsymbol{F}}$, the policy to obtain a point on the line segment between $Z_{\boldsymbol{F}^{\prime}}$ and $Z_{\boldsymbol{F}^{\prime\prime}}$ is the convex combination of $\boldsymbol{F}^{\prime}$ and $\boldsymbol{F}^{\prime\prime}$, it should have the form shown in (16). ∎

We can see that the optimal policy for the Constrained Markov Decision Process may not be deterministic. At most two elements in the policy matrix $\boldsymbol{F}$, i.e. $f_{q_{\boldsymbol{F}}(s^{*}),s^{*}}$ and $f_{q_{\boldsymbol{F}}(s^{*}),s^{*}+1}$, can be decimal, while the other elements are either 0 or 1. Policies in this form also satisfy our definition of stationary threshold-based policy at the beginning of Section III.

IV Algorithm to Efficiently Obtain the Optimal Tradeoff Curve

We design Algorithm 1 to efficiently obtain the optimal delay-power tradeoff curve and the corresponding optimal policies. Similar to [12], this algorithm takes advantage of the properties we have shown, i.e., the optimal delay-power tradeoff curve is piecewise linear, the vertices are obtained by deterministic threshold-based policies, and policies corresponding to two adjacent vertices take different actions in only one state. Therefore given the optimal policy for a certain vertex, we can narrow down the alternatives of optimal policies for its adjacent vertex. The policies corresponding to points between two adjacent vertices can also be easily generated.

Our proposed iterative algorithm starts from the bottom-right vertex of the optimal tradeoff curve, whose corresponding policy is known to transmit as much as possible. Then for each vertex we have determined, we enumerate the candidates for the next vertex. According to the properties we have obtained, we only need to search for deterministic threshold-based policies which take different actions in only one threshold. By comparing all the candidates, the next vertex will be determined by the policy candidate whose connecting line with the current vertex has the minimum absolute slope and the minimum length. Note that a vertex can be obtained by more than one policy, therefore we use lists $\mathcal{F}_{p}$ and $\mathcal{F}_{c}$ to restore all policies corresponding to the previous and the current vertices.

The complexity of this algorithm is much smaller than using general methods. Since during each iteration, one of the thresholds of the optimal policy will be decreased by 1, the maximum iteration times are $AQ$. Within each iteration, we have $A$ thresholds to try. For each candidate, the most time consuming operation, i.e. the matrix inversion, costs $\boldsymbol{O}(Q^{3})$. Therefore the complexity of the algorithm is $\boldsymbol{O}(A^{2}Q^{4})$.

In comparison, we also formulate a Linear Programming (LP) to obtain the optimal tradeoff curve. As demonstrated in [13, Chapter 11.5], all CMDP problems with infinite horizon and average cost can be formulated as Linear Programming. In our case, by taking $x_{q,s}=\pi(q)f_{q,s}$ as variables, we can formulate an LP with $QS$ variables to minimize the average delay given a certain power constraint. Due to space limitation, we provide the LP without explanations.

[TABLE]

By solving the LP, we can obtain a point on the optimal tradeoff curve. If we apply the ellipsoid algorithm to solve the LP problem, the computational complexity is $\boldsymbol{O}(S^{4}Q^{4})$. It means that, the computation to obtain one point on the optimal tradeoff curve by applying LP is larger than obtaining the entire curve with our proposed algorithm. This demonstrates the inherent advantage of using the revealed properties of the optimal tradeoff curve and the optimal policies.

V Numerical Results

In this section, we validate our theoretical results and the proposed algorithm by conducting LP numerical computation and simulations. We consider a practical scenario with adaptive M-PSK transmissions. The optional modulations are BPSK, QPSK, and 8-PSK. Assume the bandwidth = 1 MHz, the length of a timeslot = 10 ms, and the target bit error rate ber=$10^{-5}$. Assume a data packet contains 10,000 bits, and in each timeslot the number of arriving packet could be 0, 1, 2 or 3. Then by adaptively applying BPSK, QPSK, or 8-PSK, we can respectively transmit 1, 2, or 3 packets in a timeslot, which means $S=3$. Assume the one-sided noise power spectral density $N_{0}$=-150 dBm/Hz. The transmission power for different transmission rates can be calculated as $P_{0}=0$ J, $P_{1}=9.0\times 10^{-14}$ J, $P_{2}=18.2\times 10^{-14}$ J, and $P_{3}=59.5\times 10^{-14}$ J. Set the buffer size as $Q=100$.

The optimal delay-power tradeoff curves are shown in Figure 4 and Figure 5. In each figure, we vary the arrival process to get different tradeoff curves. As can be observed, the tradeoff curves generated by Algorithm 1 perfectly match the Linear Programming and simulation results. As proven in Theorem 1, the optimal tradeoff curves are piecewise linear, decreasing, and convex. The vertices of the curves obtained by Algorithm 1 are marked by squares. The corresponding optimal policies can be checked as threshold-based. The minimum average delay is 1 for all curves, because when we transmit as much as we can, all data packets will stay in the queue for exactly one timeslot. In Figure 4, with the average arrival rate increasing, the curve gets higher because of the heavier workload. In Figure 5, the three arrival processes have the same average arrival rate and different variance. When the variance gets larger, it is more likely that the queue size gets long in a short time duration, which leads to higher delay. It is interesting to characterize the effect of the variance in the arrival process, which we leave as a future work.

VI Conclusion

In this paper, we extend our previous work to obtain the optimal delay-power tradeoff and the corresponding optimal scheduling policy considering arbitrary i.i.d. arrival and adaptive transmissions. The scheduler optimize the transmission in each timeslot according to the buffer state. We formulate this problem as a CMDP, and minimize the average delay to obtain the optimal tradeoff curve. By studying the steady-state properties and the Lagrangian relaxation of the CMDP problem, we can prove that the optimal delay-power tradeoff curve is convex and piecewise linear, on which the adjacent vertices are obtained by policies taking different actions in only one state. Based on this, the optimal policies are proven to be threshold-based. We also design an efficient algorithm to obtain the optimal tradeoff curve and the optimal policies. Linear Programming and simulations are conducted to confirm the theoretical results and the proposed algorithm.

Appendix A Proof of the Equivalency to Reduce to Unichain cases

We claim that we can focus only on the unichain cases, because for any Markov process with multiple recurrent classes determined by a certain policy, we can design a policy which leads to a unichain Markov process having the same performance as any of the recurrent class. We strictly express the reason as a proposition below, and give the detailed proof.

Proposition 1**.**

In the Markov Decision Process with arbitrary arrival and adaptive transmission, if there is more than one closed communication class in the Markov chain generated by policy $\boldsymbol{F}$, which we define as $\mathcal{C}_{1}$, $\cdots$, $\mathcal{C}_{L}$ where $L>1$, then for any $1\leq l\leq L$, there exists a policy $\boldsymbol{F}_{l}$, under which the Markov chain has $\mathcal{C}_{l}$ as its only closed communication class. Furthermore, the steady-state distribution and the average cost of the Markov chain under $\boldsymbol{F}$ starting from state $c\in\mathcal{C}_{l}$ are the same as the steady-state distribution and the average cost of the Markov chain under $\boldsymbol{F}_{l}$.

Proof:

Define the set of those transient states that have access to $\mathcal{C}_{l}$ as $\mathcal{C}_{l}^{t}$. Define the set of transient states which don’t have access to $\mathcal{C}_{l}$ as $\mathcal{C}_{nl}^{t}$. Therefore $\{\mathcal{C}_{1},\cdots,\mathcal{C}_{L},\mathcal{C}_{l}^{t},\mathcal{C}_{nl}^{t}\}$ is a partition of the states of the MDP. There should exists at least one state $c\in\bigcup_{i=1,i\neq l}^{+\infty}\mathcal{C}_{i}\cup\mathcal{C}_{nl}^{t}$ which is next to a state $c^{\prime}\in\mathcal{C}_{l}\cup\mathcal{C}_{l}^{t}$. We can always change the action in state $c$ such that state $c$ can access the set $\mathcal{C}_{l}\cup\mathcal{C}_{l}^{t}$. After the modification, state $c$ will be a transient state which has access to $\mathcal{C}_{l}$. The states which communicate with $c$ will also be transient states which have access to $\mathcal{C}_{l}$.

We update the partition of states since the policy is changed. According to the above description, the set $\mathcal{C}_{l}$ won’t change, while the cardinality of $\mathcal{C}_{l}^{t}$ will be strictly increasing. Hence, by repeating the above operation for finite times, every state of the MDP will be partitioned in either $\mathcal{C}_{l}$ or $\mathcal{C}_{l}^{t}$. The Markov chain generated by the modified policy has $\mathcal{C}_{l}$ as its only closed communication class, and the modified policy is the $\boldsymbol{F}_{l}$ we request.

Since the actions of states in $\mathcal{C}_{l}$ are the same for policy $\boldsymbol{F}$ and $\boldsymbol{F}_{l}$, the steady-state distribution and the average cost corresponding to policy $\boldsymbol{F}$ starting from state $c\in\mathcal{C}_{l}$ are the same as those under policy $\boldsymbol{F}_{l}$. ∎

Appendix B Proof of Theorem 1

In order to prove Theorem 1, we will first prove a lemma showing that the mapping from $\boldsymbol{F}$ to $Z_{\boldsymbol{F}}=(P_{\boldsymbol{F}},D_{\boldsymbol{F}})$ has a partially linear property in the first subsection. In the second subsection, we will prove that the set $\mathcal{R}$ is a convex polygon, whose vertices are all obtained by deterministic scheduling policies, and the policies corresponding to adjacent vertices of $\mathcal{R}$ take different actions in only one state. In the third subsection, we will prove that the set $\mathcal{L}$ is piecewise linear, decreasing, and convex, whose vertices are obtained by deterministic scheduling policies, and the policies corresponding to adjacent vertices of $\mathcal{L}$ take different actions in only one state.

In correspondence with Theorem 1, conclusion 1) in the theorem is proven in Subsection B, conclusion 2) is proven in Subsection C, and conclusion 3) and 4) are proven by combining results in Subsection B and C.

B-A Partially Linear Property of Scheduling Policies

Lemma 1**.**

$\boldsymbol{F}$ and $\boldsymbol{F}^{\prime}$ are two policies different only when $q[n]=q$, i.e., these two matrices are different only in the $(q+1)$th row. Denote $\boldsymbol{F}^{\prime\prime}=(1-\epsilon)\boldsymbol{F}+\epsilon\boldsymbol{F}^{\prime}$ where $0\leq\epsilon\leq 1$. Then

1. There exists a certain $0\leq\epsilon^{\prime}\leq 1$ so that $P_{\boldsymbol{F}^{\prime\prime}}=(1-\epsilon^{\prime})P_{\boldsymbol{F}}+\epsilon^{\prime}P_{\boldsymbol{F}^{\prime}}$ and $D_{\boldsymbol{F}^{\prime\prime}}=(1-\epsilon^{\prime})D_{\boldsymbol{F}}+\epsilon^{\prime}D_{\boldsymbol{F}^{\prime}}$. Furthermore, parameter $\epsilon^{\prime}$ is a continuous non-decreasing function of $\epsilon$.

2. When $\epsilon$ changes from 0 to 1, point $Z_{\boldsymbol{F}^{\prime\prime}}$ moves on the line segment $\overline{Z_{\boldsymbol{F}}Z_{\boldsymbol{F}^{\prime}}}$ from $Z_{\boldsymbol{F}}$ to $Z_{\boldsymbol{F}^{\prime}}$.

Proof:

In the following, the two conclusions of the lemma will be proven one by one.

1. According to the definition of $\boldsymbol{H}_{\boldsymbol{F}}$ and $\boldsymbol{p}_{\boldsymbol{F}}$, we have that if $\boldsymbol{F}^{\prime\prime}=(1-\epsilon)\boldsymbol{F}+\epsilon\boldsymbol{F}^{\prime}$, then $\boldsymbol{H}_{\boldsymbol{F}^{\prime\prime}}=(1-\epsilon)\boldsymbol{H}_{\boldsymbol{F}}+\epsilon\boldsymbol{H}_{\boldsymbol{F}^{\prime}}$ and $\boldsymbol{p}_{\boldsymbol{F}^{\prime\prime}}=(1-\epsilon)\boldsymbol{p}_{\boldsymbol{F}}+\epsilon\boldsymbol{p}_{\boldsymbol{F}^{\prime}}$. Set $\Delta\boldsymbol{H}=\boldsymbol{H}_{\boldsymbol{F}^{\prime}}-\boldsymbol{H}_{\boldsymbol{F}}$ and $\Delta\boldsymbol{p}=\boldsymbol{p}_{\boldsymbol{F}^{\prime}}-\boldsymbol{p}_{\boldsymbol{F}}$. Since $\boldsymbol{F}$ and $\boldsymbol{F}^{\prime}$ are different only in the $(q+1)$th row, it can be derived that the $(q+1)$th column of $\Delta\boldsymbol{H}$ is the only column that can contain non-zero elements, and the $(q+1)$th element of $\Delta\boldsymbol{p}$ is its only non-zero element. Therefore $\Delta\boldsymbol{H}$ can be expressed as $\left[\boldsymbol{0},\cdots,\boldsymbol{\delta}_{q},\cdots,\boldsymbol{0}\right]$, where $\boldsymbol{\delta}_{q}$ is its $(q+1)$th column, and $\Delta\boldsymbol{p}$ can be expressed as $\left[0,\cdots,\zeta_{q},\cdots,0\right]^{T}$, where $\zeta_{q}$ is its $(q+1)$th element. Based on this, we set

[TABLE]

Hence

[TABLE]

By mathematical induction, we can prove that for $i\geq 1$,

[TABLE]

and

[TABLE]

Therefore,

[TABLE]

We have $P_{\boldsymbol{F}}=\boldsymbol{p}_{\boldsymbol{F}}^{T}\boldsymbol{H}_{\boldsymbol{F}}^{-1}\boldsymbol{c}$ and $D_{\boldsymbol{F}}=\frac{1}{E_{a}}\boldsymbol{d}^{T}\boldsymbol{H}_{\boldsymbol{F}}^{-1}\boldsymbol{c}$. Hence

[TABLE]

and

[TABLE]

Hence $\frac{P_{\boldsymbol{F}^{\prime\prime}}-P_{\boldsymbol{F}}}{P_{\boldsymbol{F}^{\prime}}-P_{\boldsymbol{F}}}=\frac{D_{\boldsymbol{F}^{\prime\prime}}-D_{\boldsymbol{F}}}{D_{\boldsymbol{F}^{\prime}}-D_{\boldsymbol{F}}}=\frac{\epsilon+\epsilon\boldsymbol{h}_{q}^{T}\boldsymbol{\delta}_{q}}{1+\epsilon\boldsymbol{h}_{q}^{T}\boldsymbol{\delta}_{q}}=\epsilon^{\prime}$, so that $P_{\boldsymbol{F}^{\prime\prime}}=(1-\epsilon^{\prime})P_{\boldsymbol{F}}+\epsilon^{\prime}P_{\boldsymbol{F}^{\prime}}$ and $D_{\boldsymbol{F}^{\prime\prime}}=(1-\epsilon^{\prime})D_{\boldsymbol{F}}+\epsilon^{\prime}D_{\boldsymbol{F}^{\prime}}$. Furthermore, it can be seen that $\epsilon^{\prime}=\frac{\epsilon+\epsilon\boldsymbol{h}_{q}^{T}\boldsymbol{\delta}_{q}}{1+\epsilon\boldsymbol{h}_{q}^{T}\boldsymbol{\delta}_{q}}$ is a continuous nondecreasing function.

2. From the first part, we proved $\frac{P_{\boldsymbol{F}^{\prime\prime}}-P_{\boldsymbol{F}}}{P_{\boldsymbol{F}^{\prime}}-P_{\boldsymbol{F}}}=\frac{D_{\boldsymbol{F}^{\prime\prime}}-D_{\boldsymbol{F}}}{D_{\boldsymbol{F}^{\prime}}-D_{\boldsymbol{F}}}=\epsilon^{\prime}$ and $\epsilon^{\prime}$ is a continuous non-decreasing function of $\epsilon$. When $\epsilon=0$, we have $\epsilon^{\prime}=0$. When $\epsilon=1$, we have $\epsilon^{\prime}=1$. Therefore when $\epsilon$ changes from 0 to 1, the point $(P_{\boldsymbol{F}^{\prime\prime}},D_{\boldsymbol{F}^{\prime\prime}})$ moves on the line segment from $(P_{\boldsymbol{F}},D_{\boldsymbol{F}})$ to $(P_{\boldsymbol{F}^{\prime}},D_{\boldsymbol{F}^{\prime}})$. The slope of the line can be expressed as

[TABLE]

∎

B-B Properties of set $\mathcal{R}$

In this subsection, we will prove that $\mathcal{R}$, the set of all feasible points in the delay-power plane, is a convex polygon whose vertices are all obtained by deterministic scheduling policies. Moreover, the policies corresponding to adjacent vertices of $\mathcal{R}$ take different actions in only one state.

Define $\mathcal{C}=\textbf{conv }\{Z_{\boldsymbol{F}}|\boldsymbol{F}\in\mathcal{F}_{D}\}$ as the convex hull of points corresponding to deterministic scheduling policies in the delay-power plane. Hence we will show that $\mathcal{R}$ is a convex polygon whose vertices are all obtained by deterministic scheduling policies by proving $\mathcal{R}=\mathcal{C}$.

The proof is made up of three parts. In Part I, we will prove $\mathcal{R}\subseteq\mathcal{C}$ by the construction method. Part II is the most difficult part. We will first define the concepts of basic polygons and compound polygons, then prove their convexity, based on which $\mathcal{R}\supseteq\mathcal{C}$ can be proven. By combining the results from Part I and II, we will have $\mathcal{R}=\mathcal{C}$. Finally, in Part III, it will be shown that policies corresponding to adjacent vertices of $\mathcal{R}$ are different in only one state.

Part I. Prove $\mathcal{R}\subseteq\mathcal{C}$

For any probabilistic policy $\boldsymbol{F}$ where $0<f_{q^{*},s^{*}}<1$, we construct

[TABLE]

and

[TABLE]

Since $0\leq\frac{f_{q,s}}{1-f_{q^{*},s^{*}}}\leq 1$, and the fact that whenever $f_{q,s}=0$, it must holds that $f^{\prime}_{q,s}=f^{\prime\prime}_{q,s}=0$, we can conclude that policies $\boldsymbol{F}^{\prime}$ and $\boldsymbol{F}^{\prime\prime}$ are feasible. It can be seen that $\boldsymbol{F}=f_{q^{*},s^{*}}\boldsymbol{F}^{\prime}+(1-f_{q^{*},s^{*}})\boldsymbol{F}^{\prime\prime}$. Since $\boldsymbol{F}$ is a convex combination of policy $\boldsymbol{F}^{\prime}$ and policy $\boldsymbol{F}^{\prime\prime}$, also $\boldsymbol{F}^{\prime}$ and $\boldsymbol{F}^{\prime\prime}$ are different only in the $(q^{*}+1)$th row, from Lemma 1, we know that $Z_{\boldsymbol{F}}$ is a convex combination of $Z_{\boldsymbol{F}^{\prime}}$ and $Z_{\boldsymbol{F}^{\prime\prime}}$. Note that $f^{\prime}_{q^{*},s^{*}}$ and $f^{\prime\prime}_{q^{*},s^{*}}$ are integers. Also, in matrices $\boldsymbol{F}^{\prime}$ and $\boldsymbol{F}^{\prime\prime}$, no new decimal elements are going to be introduced. Hence in finite steps, the point $Z_{\boldsymbol{F}}$ can be expressed as the convex combination of points corresponding to deterministic scheduling policies. That is to say $Z_{\boldsymbol{F}}\in\mathcal{C}$. From the arbitrariness of $\boldsymbol{F}$, we have $\mathcal{R}\subseteq\mathcal{C}$.

Part II. Prove $\mathcal{R}\supseteq\mathcal{C}$

In this part, we will begin with the concepts of basic polygons and compound polygons in Part II.0. Then we will prove that basic polygons and compound polygons are convex in Part II.1 and Part II.2 respectively. Based on the above results, we will prove $\mathcal{R}\supseteq\mathcal{C}$ in Part II.3.

Part II.0 The Concepts of Basic Polygons and Compound Polygons

For two deterministic policies $\boldsymbol{F}$ and $\boldsymbol{F}^{\prime}$ which are different in $K$ states, namely $q_{1},\cdots,q_{K}$, define $\boldsymbol{F}_{b_{1},b_{2},\cdots,b_{K}}(q,:)=\begin{cases}(1-b_{k})\boldsymbol{F}(q,:)+b_{k}\boldsymbol{F}^{\prime}(q,:)&q=q_{k},\\ \boldsymbol{F}(q,:)&q\neq q_{1},\cdots,q_{K},\end{cases}$ where $0\leq b_{k}\leq 1$ for all $k$. Thus $\boldsymbol{F}_{0,0,\cdots,0}=\boldsymbol{F}$, and $\boldsymbol{F}_{1,1,\cdots,1}=\boldsymbol{F}^{\prime}$. With more $b_{k}$ close to 0, the policy is more like $\boldsymbol{F}$. With more $b_{k}$ close to 1, the policy is more like $\boldsymbol{F}^{\prime}$. For policies $\boldsymbol{F}_{b_{1},\cdots,b_{k},\cdots,b_{K}}$ and $\boldsymbol{F}_{b_{1},\cdots,b_{k}^{\prime},\cdots,b_{K}}$ where $b_{k}\neq b_{k}^{\prime}$, since they are different in only one state, according to Lemma 1, the delay-power point corresponding to their convex combination $Z_{\epsilon\boldsymbol{F}_{b_{1},\cdots,b_{k},\cdots,b_{K}}+(1-\epsilon)\boldsymbol{F}_{b_{1},\cdots,b_{k}^{\prime},\cdots,b_{K}}}$ is the convex combination of $Z_{\boldsymbol{F}_{b_{1},\cdots,b_{k},\cdots,b_{K}}}$ and $Z_{\boldsymbol{F}_{b_{1},\cdots,b_{k}^{\prime},\cdots,b_{K}}}$. However, for two policies which are different in more than one state, the delay-power points corresponding to policies of their convex combination are not necessarily the convex combination of delay-power points corresponding to themselves. Therefore, we introduce the concept of generated polygon for the delay-power region of point set corresponding to policies which are convex combinations of the two policies. We plot $Z_{\boldsymbol{F}_{b_{1},\cdots,b_{K}}}$, where $b_{k}=0$ or $1$ for all $1\leq k\leq K$, and connect the points whose corresponding policies are different in only one state. Therefore any point on any line segment can be obtained by a certain policy. We define the figure as a polygon generated by $\boldsymbol{F}$ and $\boldsymbol{F}^{\prime}$. The red polygon in Figure 6a and the polygon in Figure 7a are demonstrations where $\boldsymbol{F}$ and $\boldsymbol{F}^{\prime}$ are different in 2 and 3 states respectfully. If $K=2$, we call the polygon a basic polygon. If $K>2$, we call it a compound polygon. As demonstrated in Figure 7a, a compound polygon contains multiple basic polygons.

Part II.1 Prove a Basic Polygon is Convex and Any Point Inside a Basic Polygon can be Obtained by a Policy

For better visuality, in Figure 6, we simplify the notation $Z_{\boldsymbol{F}_{b_{1},b_{2}}}$ as $b_{1},b_{2}$. By considering all possible relative positions of $Z_{\boldsymbol{F}_{0,0}}$, $Z_{\boldsymbol{F}_{0,1}}$, $Z_{\boldsymbol{F}_{1,0}}$, and $Z_{\boldsymbol{F}_{1,1}}$, there are 3 possible shapes of basic polygons in total, as shown in Figure 6a-6c respectfully. We name them as the normal shape, the boomerang shape, and the butterfly shape. The degenerate cases such as triangles, line segments and points are considered included in the above three cases. Besides $\boldsymbol{F}_{b_{1},b_{2}}$ with integral $b_{1},b_{2}$ and the line segments connecting them, in the figures we also plot the points corresponding to policy $\boldsymbol{F}_{b_{1},b_{2}}$ where one of $b_{1},b_{2}$ is integer and the other one is decimal. We connect the points corresponding to policies which have the same $b_{1}$ or $b_{2}$ with dashed lines. As demonstrated in Figure 6, we draw line segments $\overline{Z_{\boldsymbol{F}_{b_{1},0}}Z_{\boldsymbol{F}_{b_{1},1}}}$ where $b_{1}=0.1,0.2,\cdots,0.9$ and $\overline{Z_{\boldsymbol{F}_{0,b_{2}}}Z_{\boldsymbol{F}_{1,b_{2}}}}$ where $b_{2}=0.1,0.2,\cdots,0.9$. For any specific $b_{1}$ and $b_{2}$, the point $Z_{\boldsymbol{F}_{b_{1},b_{2}}}$ should be on both $\overline{Z_{\boldsymbol{F}_{b_{1},0}}Z_{\boldsymbol{F}_{b_{1},1}}}$ and $\overline{Z_{\boldsymbol{F}_{0,b_{2}}}Z_{\boldsymbol{F}_{1,b_{2}}}}$. Because of the existence of $Z_{\boldsymbol{F}_{b_{1},b_{2}}}$, line segments $\overline{Z_{\boldsymbol{F}_{b_{1},0}}Z_{\boldsymbol{F}_{b_{1},1}}}$ and $\overline{Z_{\boldsymbol{F}_{0,b_{2}}}Z_{\boldsymbol{F}_{1,b_{2}}}}$ should always have an intersection point for any specific $b_{1}$ and $b_{2}$. However, if there exist line segments outside the polygon, there exist $b_{1}$ and $b_{2}$ whose line segments don’t intersect. Therefore, in the boomerang shape, there will always exist $b_{1}$ and $b_{2}$ whose line segments don’t intersect. In the butterfly shape, there will exist $b_{1}$ and $b_{2}$ whose line segments don’t intersect except the case that all the line segments are inside the basic polygon, as shown in Figure 6d, which is named as the slender butterfly shape. In the slender butterfly shape, there exists a specific $b_{1}^{*}$ such that $\overline{Z_{\boldsymbol{F}_{b_{1}^{*},0}}Z_{\boldsymbol{F}_{b_{1}^{*},1}}}$ degenerates into a point, or there exists a specific $b_{2}^{*}$ such that $\overline{Z_{\boldsymbol{F}_{0,b_{2}^{*}}}Z_{\boldsymbol{F}_{1,b_{2}^{*}}}}$ degenerates into a point. Without loss of generality, we assume it is the $b_{1}^{*}$ case. It means that under policy $\boldsymbol{F}_{b_{1}^{*},b_{2}}$, state $q_{2}$, the state corresponding to $b_{2}$, is a transient state. For $b_{1}\in(b_{1}^{*}-\epsilon,b_{1}^{*}+\epsilon)$ when $\epsilon$ is small enough, the Markov chain applying policy $F_{b_{1},b_{2}}$ also has $q_{2}$ as a transient state, therefore $\overline{Z_{\boldsymbol{F}_{b_{1},0}}Z_{\boldsymbol{F}_{b_{1},1}}}$ also degenerates into a point. Thus $\overline{Z_{\boldsymbol{F}_{0,0}}Z_{\boldsymbol{F}_{1,0}}}$ and $\overline{Z_{\boldsymbol{F}_{0,1}}Z_{\boldsymbol{F}_{1,1}}}$ overlap, which means the slender butterfly shape always degenerates to a line segment, which can also be considered as a normal shape. Since the normal shape is the only possible shape of a basic polygon, the basic polygon is convex. Since the transition from the point $Z_{\boldsymbol{F}_{0,0}}$ to $Z_{\boldsymbol{F}_{1,1}}$ is termwise monotone and continuous, every point inside the basic polygon can be obtained by a policy.

Part II.2 Prove a Compound Polygon is Convex

For any two deterministic policies $\boldsymbol{F}$ and $\boldsymbol{F}^{\prime}$, if the compound polygon generated by them is not convex, then there must exist two vertices whose connecting line is outside the compound polygon, as demonstrated by $\overline{Z_{1}Z_{2}}$ in Figure 7b. Therefore, there must also exist two vertices who are connecting to the same point and their connecting line is outside the compound polygon, as demonstrated by $\overline{Z_{1}Z_{3}}$. The policies corresponding to these two vertices must be different in only two states based on previous conclusions, therefore there must be a basic polygon generated by them, as demonstrated in Figure 7b by the filled polygon. Since $\overline{Z_{1}Z_{3}}$ is outside the compound polygon, it is for sure that $\overline{Z_{1}Z_{3}}$ is outside the basic polygon too. This is impossible because basic polygons are always convex. Hence we can conclude that all generated compound polygons are convex.

Part II.3 Prove $\mathcal{R}\supseteq\mathcal{C}$

For arbitrary point $C\in\mathcal{C}$, it will fall into one of the compound polygons, because otherwise, there will be at least one point corresponding to a deterministic policy outside any compound polygons. All compound polygons can be covered by basic polygons, therefore $C$ is inside at least one basic polygon. Since any point inside a basic polygon can be obtained by a policy, the point $C\in\mathcal{R}$. From the arbitrariness of $C$, we have $\mathcal{R}\supseteq\mathcal{C}$.

From Part II.1 and Part II.2, it can be proven that $\mathcal{R}=\mathcal{C}$. Since there are only finite deterministic policies in total, the set $\mathcal{R}$ is a convex polygon with its vertices all obtained by deterministic scheduling policies.

Part III. Adjacent Vertices of $\mathcal{R}$

For any two adjacent vertices $Z_{\boldsymbol{F}}$ and $Z_{\boldsymbol{F}^{\prime}}$ of $\mathcal{R}$, if $\boldsymbol{F}$ and $\boldsymbol{F}^{\prime}$ are different in more than one state, the polygon generated by them is convex. If the line segment $\overline{Z_{\boldsymbol{F}}Z_{\boldsymbol{F}^{\prime}}}$ is inside the generated polygon, $Z_{\boldsymbol{F}}$ and $Z_{\boldsymbol{F}^{\prime}}$ are impossible to be adjacent. If the line segment $\overline{Z_{\boldsymbol{F}}Z_{\boldsymbol{F}^{\prime}}}$ is on the boundary of the generated polygon, there will be other vertices between them, then $Z_{\boldsymbol{F}}$ and $Z_{\boldsymbol{F}^{\prime}}$ are still not adjacent. Therefore, we can conclude that policies $\boldsymbol{F}$ and $\boldsymbol{F}^{\prime}$ are deterministic and different in only one state.

B-C Properties of set $\mathcal{L}$

In this subsection, we will prove that the optimal delay-power tradeoff curve $\mathcal{L}$ is piecewise linear, decreasing, and convex. The vertices of the curve are obtained by deterministic scheduling policies. Moreover, the policies corresponding to adjacent vertices of $\mathcal{L}$ take different actions in only one state.

Proof:

Monotonicity:

Since $\mathcal{L}=\{(P,D)\in\mathcal{R}|\forall(P^{\prime},D^{\prime})\in\mathcal{R},\text{ either }P^{\prime}\geq P\text{ or }D^{\prime}\geq D\}$, for any $(P_{1},D_{1}),(P_{2},D_{2})\in\mathcal{L}$ where $P_{1}<P_{2}$, we should have $D_{1}\geq D_{2}$. Therefore $\mathcal{L}$ is decreasing.

Convexity:

Since $\mathcal{R}$ is a convex polygon, for any $(P_{1},D_{1}),(P_{2},D_{2})\in\mathcal{L}$, their convex combination is $(\theta P_{1}+(1-\theta)P_{2},\theta D_{1}+(1-\theta)D_{2})\in\mathcal{R}$. Hence there exists a point $(P_{\theta},D_{\theta})$ on $\mathcal{L}$ where $P_{\theta}=\theta P_{1}+(1-\theta)P_{2}$, and $D_{\theta}\leq\theta D_{1}+(1-\theta)D_{2}$. Therefore $\mathcal{L}$ is convex.

Piecewise Linearity:

Since $\mathcal{R}$ is a convex polygon, it can be expressed as the intersection of a finite number of halfspaces, i.e., $\mathcal{R}=\bigcap_{i=1}^{I}\{(P,D)|a_{i}P+b_{i}D\geq c_{i}\}$. We divide $(a_{i},b_{i},c_{i})$ into 2 categories according to the value of $a_{i}$ and $b_{i}$ as $(a_{i}^{+},b_{i}^{+},c_{i}^{+})$ for $i=1,\cdots,I^{+}$ if $a_{i}>0$ and $b_{i}>0$, and $(a_{i}^{-},b_{i}^{-},c_{i}^{-})$ for $i=1,\cdots,I^{-}$ if $a_{i}\leq 0$ or $b_{i}\leq 0$. We have $I=I^{+}+I^{-}$ and $I^{+},I^{-}>0$. Then $\mathcal{R}=\bigcap_{i=1}^{I^{+}}\{(P,D)|a_{i}^{+}P+b_{i}^{+}D\geq c_{i}^{+}\}\cap\bigcap_{i=1}^{I^{-}}\{(P,D)|a_{i}^{-}P+b_{i}^{-}D\geq c_{i}^{-}\}$. For $1\leq l\leq I^{+}$, define $\mathcal{L}_{l}=\{(P,D)|a_{l}^{+}P+b_{l}^{+}D=c_{l}^{+}\}\cap\bigcap_{i=1,i\neq l}^{I^{+}}\{(P,D)|a_{i}^{+}P+b_{i}^{+}D\geq c_{i}^{+}\}\cap\bigcap_{i=1}^{I^{-}}\{(P,D)|a_{i}^{-}P+b_{i}^{-}D\geq c_{i}^{-}\}$.

For all $(P,D)\in\mathcal{L}_{l}$, we have $(P,D)\in\mathcal{R}$. For all $(P^{\prime},D^{\prime})\in\mathcal{R}$, since $a_{l}^{+}P^{\prime}+b_{l}^{+}D^{\prime}\geq c_{l}^{+}=a_{l}^{+}P+b_{l}^{+}D$, it should hold that $P^{\prime}\geq P$ or $D^{\prime}\geq D$. According to the definition of $\mathcal{L}$, we have $(P,D)\in\mathcal{L}$. Therefore $\mathcal{L}_{l}\subseteq\mathcal{L}$.

For all $(P,D)\in\mathcal{L}$, we consider three cases: 1) If $a_{i}^{+}P+b_{i}^{+}D>c_{i}^{+}$ for all $1\leq i\leq I^{+}$ and $a_{i}^{-}P+b_{i}^{-}D>c_{i}^{-}$ for all $b_{i}^{-}>0$, set $\epsilon=\min_{b_{i}>0}\frac{a_{i}P+b_{i}D-c_{i}}{b_{i}}$ so that $a_{i}P+b_{i}(D-\epsilon)\geq c_{i}$ for all $b_{i}>0$. Since $(P,D)\in\mathcal{R}$, for all $b_{i}\leq 0$ $a_{i}P+b_{i}D\geq c_{i}$, therefore $a_{i}P+b_{i}(D-\epsilon)\geq c_{i}$ for all $b_{i}\leq 0$. Hence $(P,D-\epsilon)\in\mathcal{R}$, which is against the definition of $\mathcal{L}$. 2) If $a_{i}^{+}P+b_{i}^{+}D>c_{i}^{+}$ for all $1\leq i\leq I^{+}$ and $a_{i}^{-}P+b_{i}^{-}D>c_{i}^{-}$ for all $a_{i}^{-}>0$, set $\epsilon=\min_{a_{i}>0}\frac{a_{i}P+b_{i}D-c_{i}}{a_{i}}$ so that $a_{i}(P-\epsilon)+b_{i}D\geq c_{i}$ for all $a_{i}>0$. Since $(P,D)\in\mathcal{R}$, for all $a_{i}\leq 0$ $a_{i}P+b_{i}D\geq c_{i}$, therefore $a_{i}(P-\epsilon)+b_{i}D\geq c_{i}$ for all $a_{i}\leq 0$. Hence $(P-\epsilon,D)\in\mathcal{R}$, which is against the definition of $\mathcal{L}$. 3) If $a_{i}^{+}P+b_{i}^{+}D>c_{i}^{+}$ for all $1\leq i\leq I^{+}$, and there exists $i^{*}$ and $j^{*}$ such that $a_{i^{*}}^{-}\leq 0$, $b_{i^{*}}^{-}>0$, $a_{j^{*}}^{-}>0$, $b_{j^{*}}^{-}\leq 0$, $a_{i^{*}}^{-}P+b_{i^{*}}^{-}D=c_{i^{*}}^{-}$, $a_{j^{*}}^{-}P+b_{j^{*}}^{-}D=c_{j^{*}}^{-}$. For all $(P^{\prime},D^{\prime})\in\mathcal{R}$, either $P^{\prime}\geq P$, $D^{\prime}\geq D$ or $P^{\prime}\leq P$, $D^{\prime}\leq D$. If there exists $P^{\prime}<P$ and $D^{\prime}<D$, then $(P,D)$ is against the definition of $\mathcal{L}$. If $P^{\prime}\leq P$ and $D^{\prime}\leq D$ for all $(P^{\prime},D^{\prime})$, since for all $1\leq i\leq I^{+}$, we have $a_{i}^{+}P+b_{i}^{+}D>c_{i}^{+}$, therefore $a_{i}^{+}P^{\prime}+b_{i}^{+}D^{\prime}>c_{i}^{+}$. Hence $\mathcal{L}_{i}\cap\mathcal{R}=\emptyset$, which is against the condition. From the above three cases, for all $(P,D)\in\mathcal{L}$, there exists at least one certain $l^{*}$ such that $a_{l^{*}}^{+}P+b_{l^{*}}^{+}D=c_{l^{*}}^{+}$, which means $(P,D)\in\mathcal{L}_{l^{*}}$.

From above we can see that $\mathcal{L}=\bigcup_{l=1}^{I^{+}}\mathcal{L}_{l}$. Therefore $\mathcal{L}$ is piecewise linear.

Properties of Vertices of $\mathcal{L}$:

The vertices of $\mathcal{L}$ are also the vertices of $\mathcal{R}$, and adjacent vertices of $\mathcal{L}$ are also adjacent vertices of $\mathcal{R}$. From the results in Section B, vertices of $\mathcal{L}$ are obtained by deterministic scheduling policies, and the policies corresponding to adjacent vertices of $\mathcal{L}$ are different in only one state. ∎

Appendix C Proof of Theorem 2

Proof:

From the literature, it is proven that there exists an optimal deterministic stationary policy. Therefore, in the proof, we only consider deterministic policies. Let $s(q)$ denote the transmitting packet number when $q[n]=q$. Define

[TABLE]

In the following, we will apply a nested induction method to prove the theorem, which utilizes the policy iteration algorithm for Markov Decision Processes. For a Markov Decision Process considering an average cost, the policy iteration algorithm, which is shown in Algorithm 2, can always converge to the optimal scheduling policy in finite steps, which is proven in [14, Theorem 8.6.6] and [15, Proposition 3.4]. In the algorithm, the function $h^{(m)}(q)$ will in the final converge to $h(q)$, which is normally known as the potential function or the bias function of the Markov Decision Process. The function $h(q)$ can be interpreted as the expected total difference between the cost starting from a specific state and the stationary cost.

We sketch the proof as follows. Initially, we assign $h^{(0)}(q)$ as a strictly convex function in $q$. Then in Part I, it will be demonstrated by the mathematical induction method that, in the policy improvement step of the policy iteration algorithm, for any $m$, if $h^{(m)}(q)$ is strictly convex in $q$, then $s^{(m+1)}(q)$ has the threshold-based property. On the other hand, in Part II, we show that in the policy evaluation step of the policy iteration algorithm, if $s^{(m+1)}(q)$ has the threshold-based property, then $h^{(m+1)}(q)$ is strictly convex in $q$. Based on the above derivations, we can prove the required conclusion by mathematical induction.

Part I. The Policy Improvement Step: Convexity of $h^{(m)}(q)$ in $q$ $\rightarrow$ threshold-based property of $s^{(m+1)}(q)$

Assume $h^{(m)}(q)$ is strictly convex in $q$. In the following, we will show that $s^{(m+1)}(q)$ has the threshold-based property.

For a feasible policy, we have $s^{(m+1)}(0)=0$, and $s^{(m+1)}(1)=0$ or $1$. Therefore $s^{(m+1)}(q+1)-s^{(m+1)}(q)=0$ or $1$ when $q=0$. 2. 2.

Define $s_{1}=s^{(m+1)}(q_{1})$ for a specific $q_{1}$. According to the Policy Improvement step, we have inequalities

[TABLE]

Since $h^{(m)}(q)$ is strictly convex in $q$, we have

[TABLE]

Since $P_{s}$ is strictly convex, we have

[TABLE]

From (60) and (62), we can obtain that

[TABLE]

From (61) and (63), we can obtain that

[TABLE]

From (64) and (65), it is shown that $s^{(m+1)}(q_{1}+1)$ can only be $s_{1}$ or $s_{1}+1$. That is to say, we have $s^{(m+1)}(q_{1}+1)-s^{(m+1)}(q_{1})=0$ or $1$.

From the above derivations, we prove by mathematical induction that $s^{(m+1)}(q)$ has the threshold-based property.

Part II. The Policy Evaluation Step: Threshold-based property of $s^{(m+1)}(q)$ $\rightarrow$ convexity of $h^{(m+1)}(q)$ in $q$

Assume $s^{(m+1)}(q)$ has the threshold-based property. We continue to use the same notation as in Part I, i.e., define $s_{1}=s^{(m+1)}(q_{1})$ for a specific $q_{1}$, and $s^{(m+1)}(q_{1}+1)=s_{1}$ or $s_{1}+1$.

If $s^{(m+1)}(q_{1}+1)=s_{1}$,

[TABLE]

On the other hand,

[TABLE] 2. 2.

If $s^{(m+1)}(q_{1}+1)=s_{1}+1$,

[TABLE]

In conclusion, for any specific $q_{1}$, we have

[TABLE]

That is to say, $h^{(m+1)}(q+1)-h^{(m+1)}(q)$ is strictly increasing. Therefore, we have $h^{(m+1)}(q)$ is strictly convex in $q$.

Based on our assumption for the initial value $h^{(0)}(q)$, as well as the derivations in Part I and II, it is proven by mathematical induction that $s^{(m)}(q)$ holds the threshold-based property for all $m\geq 1$. Since $s^{(m)}(q)$ converges to the optimal policy $s(q)$ for sure in finite steps, the optimal policy $s(q)$ will also hold the threshold-based property. ∎