Optimal Network Control in Partially-Controllable Networks

TL;DR

This paper develops new algorithms for optimal network control in partially-controllable networks, addressing unknown policies of uncontrollable nodes and balancing throughput, delay, and convergence.

Contribution

It introduces TMW, a low-complexity algorithm for queue-agnostic policies, and TUCRL, a reinforcement learning method for queue-dependent policies, both tailored for partially-controllable networks.

Findings

TMW is throughput-optimal in queue-agnostic scenarios.

TUCRL achieves tunable tradeoffs among throughput, delay, and convergence.

The algorithms perform effectively in models with unknown uncontrollable node policies.

Abstract

The effectiveness of many optimal network control algorithms (e.g., BackPressure) relies on the premise that all of the nodes are fully controllable. However, these algorithms may yield poor performance in a partially-controllable network where a subset of nodes are uncontrollable and use some unknown policy. Such a partially-controllable model is of increasing importance in real-world networked systems such as overlay-underlay networks. In this paper, we design optimal network control algorithms that can stabilize a partially-controllable network. We first study the scenario where uncontrollable nodes use a queue-agnostic policy, and propose a low-complexity throughput-optimal algorithm, called Tracking-MaxWeight (TMW), which enhances the original MaxWeight algorithm with an explicit learning of the policy used by uncontrollable nodes. Next, we investigate the scenario where uncontrollable nodes use a queue-dependent policy and the problem is formulated as an MDP with unknown queueing dynamics. We propose a new reinforcement learning algorithm, called Truncated Upper Confidence Reinforcement Learning (TUCRL), and prove that TUCRL achieves tunable three-way tradeoffs between throughput, delay and convergence rate.

Equations

$$\begin{split}\small&Q_{ik}(t+1)\\ =&Q_{ik}(t)+a_{ik}(t)+\sum_{j\in\mathcal{N}}\widetilde{f}_{jik}(t)-\sum_{j\in\mathcal{N}}\widetilde{f}_{ijk}(t)\\ \leq&\Big{[}Q_{ik}(t)+a_{ik}(t)+\sum_{j\in\mathcal{N}}f_{jik}(t)-\sum_{j\in\mathcal{N}}f_{ijk}(t)\Big{]}^{+},\end{split}$$

$$0\le a_{ik}(t)\le D,0\le f_{ijk}(t)\le D,\forall i,j,k.$$

$$\underset{t\to \infty }{\lim }\frac{\mathbb{E}[Q_{ik}(t)]}{t}=0,\forall i,k.$$

$${\Delta }_{ijk}(t)=g_{ijk}(t)-{\tilde{f}}_{ijk}(t),\forall i\in \mathcal{U},$$

$$\small\begin{split}\min_{\mathbf{g}(t)\in\mathcal{F}_{\omega_{t}}}~{}~{}~{}&\sum_{i,k}X_{ik}(t)\Big{[}a_{ik}(t)+\sum_{j}g_{jik}(t)-\sum_{j}g_{ijk}(t)\Big{]}\\ &+\sum_{i,j,k}Y_{ijk}(t)\Big{(}g_{ijk}(t)-\widetilde{f}_{ijk}(t)\Big{)},\end{split}$$

$$\underset{\mathbf{g}(t)\in {\mathcal{F}}_{{\omega }_t}}{\max }\sum _{(i,j)}\sum _k{g}_{ijk}(t)W_{ijk}(t),$$

$$W_{ijk}(t)=X_{ik}(t)-X_{jk}(t)-Y_{ijk}(t).$$

$$\small\begin{split}&X_{ik}(t+1)=\Big{[}X_{ik}(t)+a_{ik}(t)+\sum_{j\in\mathcal{N}}g_{jik}(t)-\sum_{j\in\mathcal{N}}g_{ijk}(t)\Big{]}^{+}\\ &Y_{ijk}(t+1)=Y_{ijk}(t)+\Delta_{ijk}(t)\end{split}$$

$${\lambda }_{ik}+\sum _{j\in \mathcal{N}}{\tilde{f}}_{jik}-\sum _{j\in \mathcal{N}}{\tilde{f}}_{ijk}\le 0,\forall i,k,$$

$${\tilde{f}}_{ijk}=\underset{T\to \infty }{\lim }\frac{1}{T}\sum _{t=0}^{T-1}\mathbb{E}[{\tilde{f}}_{ijk}(Q_{ik}^{\ast }(t))]$$

$$\begin{split}\small Q_{ik}(t+1)&\leq\Big{[}Q_{ik}(t)+a_{ik}(t)+\sum_{j\in\mathcal{C}}f_{jik}(t)\\ &~{}~{}~{}~{}+\sum_{j\in\mathcal{U}}f_{jik}(t)-\sum_{j\in\mathcal{N}}f_{ijk}(t)\Big{]}^{+},~{}\forall i,k.\end{split}$$

$$\mathbf{Q}(t+1)=\beta ({\mathbf{f}}^c(t),\mathbf{Q}(t),{\omega }_t),$$

$$J^{\pi}\Big{(}M,\mathbf{Q}(0)\Big{)}=\lim_{T\rightarrow\infty}\frac{1}{T}\sum_{t=0}^{T-1}\mathbb{E}_{\pi,M}\Big{[}\sum_{i,k}Q_{ik}(t)\Big{|}\mathbf{Q}(0)\Big{]},$$

$$\hat{P}({\mathbf{Q}}^{\prime }|\mathbf{Q},\alpha )=\frac{n_{\ell }(\mathbf{Q},\alpha ,{\mathbf{Q}}^{\prime })}{n_{\ell }(\mathbf{Q},\alpha )},$$

$$\small\mathcal{P}_{\ell}=\Big{\{}P:\Big{|}\Big{|}P(\cdot|\mathbf{Q},\alpha)-\hat{P}(\cdot|\mathbf{Q},\alpha)\Big{|}\Big{|}_{1}\leq\sqrt{\frac{C\log(2|\mathcal{A}|t_{\ell}V)}{\max\{1,n_{\ell}(\mathbf{Q},\alpha)\}}}\Big{\}}.$$

$$\lim_{T\rightarrow\infty}\frac{1}{T}\sum_{t=0}^{T-1}\sum_{i,k}\mathbb{E}[Q_{ik}(t)]\leq\Theta(1)-\Theta\Big{(}\frac{1}{V}\Big{)}.$$

$$\lim_{T\rightarrow\infty}\mathbb{E}[\eta_{T}]\leq\Theta\Big{(}\frac{1}{V}\Big{)},$$

$$({\mu }_{24}(t),{\mu }_{34}(t))=\{\begin{array}{ll}(0.5,0)& Q_3(t)\le 10\\ (0,1)& Q_2(t)\le 10\text{ and }{Q}_3(t)>10\\ (0.25,0.25)& Q_2(t)>10\text{ and }{Q}_3(t)>10\end{array}$$

$${\lambda }_{ik}+\sum _{j\in \mathcal{N}}{\mu }_{jik}-\sum _{j\in \mathcal{N}}{\mu }_{ijk}\le 0,\forall i,k$$

$${\mu }_{ijk}-{\tilde{f}}_{ijk}=0,\forall i,j,k,$$

$$r_{ijk}(t)={\tilde{f}}_{ijk}(Q_{ik}^{\ast }(t)),\forall t,\forall i,j,k.$$

$$\mathbb{E}[{\mu }_{ijk}(t)]=\underset{T\to \infty }{\lim }\frac{1}{T}\sum _{t=0}^{T-1}\mathbb{E}[r_{ijk}(t)],\forall i,j,k.$$

$${\mu }_{ijk}=\mathbb{E}[{\mu }_{ijk}(t)]=\underset{T\to \infty }{\lim }\frac{1}{T}\sum _{t=0}^{T-1}\mathbb{E}[r_{ijk}(t)]={\tilde{f}}_{ijk},\forall i,j,k.$$

$${\lambda }_{ik}+\sum _{j\in \mathcal{N}}{\mu }_{jik}-\sum _{j\in \mathcal{N}}{\mu }_{ijk}\le 0,\forall i,k,$$

$$\underset{t\to \infty }{\lim }\frac{{\sum }_{i,k}\mathbb{E}[X_{ik}(t)]}{t}=0,$$

$$\underset{t\to \infty }{\lim }\frac{{\sum }_{i,j,k}\mathbb{E}[|Y_{ijk}(t)|]}{t}=0.$$

$$\Phi (t)=\sum _{i,k}{X}_{ik}^2(t)+\sum _{i,j,k}{Y}_{ijk}^2(t).$$

$$\sum_{i,k}X_{ik}^{2}(t+1)-\sum_{i,k}X_{ik}^{2}(t)\leq\sum_{i,k}2X_{ik}(t)\Big{(}a_{ik}(t)+\sum_{j\in\mathcal{N}}g_{jik}(t)-\sum_{j\in\mathcal{N}}g_{ijk}(t)\Big{)}+N^{3}KD^{2},$$

$$\sum_{i,j,k}Y^{2}_{ijk}(t+1)-\sum_{i,j,k}Y^{2}_{ijk}(t)\leq\sum_{i,j,k}2Y_{ijk}(t)\Delta_{ijk}\Big{(}g_{ijk}(t),f_{ijk}(t),Q_{ik}(t)\Big{)}+N^{2}KD^{2},$$

$$\begin{split}\delta(t)&\leq 2\mathbb{E}\Big{[}\sum_{i,k}X_{ik}(t)\Big{(}a_{ik}(t)+\sum_{j\in\mathcal{N}}g_{jik}(t)-\sum_{j\in\mathcal{N}}g_{ijk}(t)\Big{)}+\sum_{i,j,k}Y_{ijk}(t)\Delta_{ijk}\Big{(}g_{ijk}(t),f_{ijk}(t),Q_{ik}(t)\Big{)}\Big{]}+2N^{3}KD^{2}\\ &\leq 2\mathbb{E}\Big{[}\sum_{i,k}X_{ik}(t)\Big{(}a_{ik}(t)+\sum_{j\in\mathcal{N}}\mu_{jik}(t)-\sum_{j\in\mathcal{N}}\mu_{ijk}(t)\Big{)}+\sum_{i,j,k}Y_{ijk}(t)\Delta_{ijk}\Big{(}\mu_{ijk}(t),f_{ijk}(t),Q_{ik}(t)\Big{)}\Big{]}+2N^{3}KD^{2},\end{split}$$

Full text

Technical Report

I Introduction

Optimal network control has been an active area of research for more than thirty years, and many efficient routing algorithms have been developed over the past few decades, such as the well-known throughput-optimal BackPressure routing algorithm [26]. The effectiveness of these algorithms usually relies on the premise that all of the nodes in a network are fully controllable. Unfortunately, an increasing number of real-world networked systems are only partially controllable, where a subset of nodes are not managed by the network operator and use some unknown network control policy, such as overlay-underlay networks.

An overlay-underlay network consists of overlay nodes and underlay nodes [3, 16, 19]. The overlay nodes can implement state-of-the-art algorithms while the underlay nodes are uncontrollable and use some unknown protocols (e.g., legacy protocols). Figure 1 shows an overlay-underlay network where the communications among overlay nodes rely on the uncontrollable underlay nodes. Overlay networks have been used to improve the capabilities of computer networks for a long time (e.g., content delivery [21]).

Due to the unknown behavior of uncontrollable nodes, the existing routing algorithms may yield poor performance in a partially-controllable network. For example, Figure 2 shows an example where the well-known Backpressure routing algorithm [26] fails to deliver the maximum throughput when some nodes are uncontrollable. In particular, uncontrollable node 3 adopts a policy that does not preserve the flow conservation law such that its backlog builds up, but uncontrollable node 2 hides this backlog information from node 1. As a result, if node 1 uses Backpressure routing, it always transmits packets to node 2, although these packets will never be delivered. A smarter algorithm should be able to learn the behavior of the uncontrollable nodes such that node 1 only sends packets along route $1\rightarrow 5\rightarrow 4$.

As a result, it is important to develop new network control algorithms that achieve consistently good performance in a partially-controllable environment. In this paper, we study efficient network control algorithms that can stabilize a partially-controllable network whenever possible. In particular, we consider two scenarios.

First, we investigate the scenario where uncontrollable nodes use a queue-agnostic policy, which captures a wide range of practical network protocols, such as shortest path routing (e.g., OSPF, RIP), multi-path routing (e.g., ECMP) and randomized routing algorithms. In this scenario, we propose a low-complexity throughput-optimal algorithm, called Tracking-MaxWeight (TMW), which enhances the original MaxWeight algorithm [26] with an explicit learning of the policy used by uncontrollable nodes.

Second, we study the scenario where uncontrollable nodes use a queue-dependent policy, i.e., the action taken by uncontrollable nodes relies on the observed queue length vector (e.g., Backpressure routing). In this scenario, we show that the queueing dynamics become unknown and no longer follow the classic Lindley recursion [6], which makes the problem fundamentally different from the traditional network optimization framework: we not only need to know how to perform optimal network control but also need to learn the queueing dynamics in an efficient way. We formulate the problem as a Markov Decision Process (MDP) with unknown dynamics, and propose a new reinforcement learning algorithm, called Truncated Upper Confidence Reinforcement Learning (TUCRL), that is shown to achieve network stability under mild conditions.

I-A Related Work

Most of the existing works on network optimization in a partially-controllable environment are in the context of overlay-underlay networks. An important feature of overlay-underlay networks is that underlay nodes are not controllable and may adopt arbitrary (unknown) policies. The objective is to find efficient control policies for the controllable overlay nodes in order to optimize certain performance metrics (e.g., throughput). In [3], the authors showed that the well-known BackPressure algorithm [26], which was shown to be throughput-optimal in a wide range of scenarios, may lead to a loss in throughput when used in an overlay-underlay setting, and proposed a heuristic routing algorithm for overlay nodes called Overlay Backpressure Policy (OBP). An optimal backpressure-type routing algorithm for a special case, where the underlay paths do not overlap with each other, was given in [16]. Recently, [19] showed that the overlay routing algorithms proposed in [3][16] are not throughput-optimal in general, and developed the Optimal Overlay Routing Policy (OORP) for overlay nodes. However, all of the existing overlay routing algorithms [3, 16, 19] impose very stringent assumptions about the behavior of underlay nodes. In particular, the underlay nodes are required to use fixed-path routing (e.g., shortest-path routing) and maintain stability whenever possible, which fails to account for many important underlay policies (e.g., underlay nodes may use multi-path routing).

In terms of technical tools, our work leverages techniques from reinforcement learning, since a partially-controllable network with queue-dependent uncontrollable policy can be formulated as an MDP with unknown dynamics. Over the past few decades, many reinforcement learning algorithms have been developed, such as Q-learning [22], actor critic [4] and policy gradient [28]. Recently, the successful applications of deep neural networks in reinforcement learning algorithms have produced many deep reinforcement learning algorithms such as DQN [8], DDPG [5] and TRPO [20]. However, most of these methods are heuristic-based and do not have any performance guarantees. Among the existing reinforcement learning algorithms there are a few that do provide good performance bounds, such as model-based reinforcement learning algorithms UCRL [1, 2] and PSRL [15, 13]. Unfortunately, these algorithms require that the size of the state space be relatively small, which cannot be directly applied in our context since the state space (i.e., queue length space) contains countably-infinite states. In this work, we combine the UCRL algorithm with a queue truncation technique and propose the Truncated Upper Confidence Reinforcement Learning (TUCRL) algorithm that has good performance guarantees even with countably-infinite queue length space.

I-B Our Contributions

In this paper, we investigate optimal network control for a partially-controllable network. Whereas existing works (e.g., [3, 16, 19]) imposed very stringent assumptions about the behavior of uncontrollable nodes for analytical tractability, this is the first work that establishes stability results under the generalized partially-controllable network model. In particular, we develop two network control algorithms.

First, we develop a low-complexity Tracking-MaxWeight (TMW) algorithm that is guaranteed to achieve network stability if uncontrollable nodes adopt queue-agnostic policies. The Tracking-MaxWeight algorithm enhances the original MaxWeight algorithm [26] with an explict learning of the policy used by uncontrollable nodes.

Next, we propose a new reinforcement learning algorithm (i.e., the TUCRL algorithm) in the more challenging scenario where uncontrollable nodes may use queue-dependent policies. It combines the state-of-art model-based UCRL algorithm [1, 2] with a queue truncation technique to overcome the problem with countably-infinite queue length space. We prove that TUCRL achieves network stability by dropping a negligible fraction of packets. We also show that the TUCRL algorithm maintains a three-way tradeoff between delay, throughput and convergence rate.

II System Model

Consider a networked system with $N$ nodes (the set of all nodes is denoted by $\mathcal{N}$). There are $K$ flows in the network and each node $i$ maintains a queue for buffering undelivered packets for each flow $k$. As a result, there are $NK$ queues in the network, and we denote by $\mathbf{Q}(t)$ the queue length vector at the beginning of time slot $t$, where its element $Q_{ik}(t)$ represents the queue length for flow $k$ at node $i$.

Let $\omega_{t}$ be the network event that occurs in slot $t$, which includes information about the current network parameters, such as a vector of channel conditions for each link and a vector of exogenous arrivals to each queue. We assume that the sequence of network events $\{\omega_{t}\}_{t\geq 0}$ follow a stationary stochastic process. In particular, the vector of exogenous packet arrivals is denoted by $\mathbf{a}(\omega_{t})=\{a_{ik}(t)\}_{i,k}$, where $a_{ik}(t)$ is the number of exogenous arrivals to queue $(i,k)$ in slot $t$. Denote by $\lambda_{ik}=\mathbb{E}[a_{ik}(t)]$ the expected exogenous packet arrival rate to queue $(i,k)$ in steady state.

At the beginning of each time slot $t$, after observing the current network event $\omega_{t}$ and the current queue length vector $\mathbf{Q}(t)$, each node $i$ needs to make a routing decision $f_{ijk}(t)$ indicating the offered transmission rate for flow $k$ over link $i\rightarrow j$. The corresponding network routing vector is denoted by $\mathbf{f}(t)=\{f_{ijk}(t)\}_{i,j,k}$.

There are two types of nodes in the network: controllable nodes (the set of controllable nodes is denoted by $\mathcal{C}$) and uncontrollable nodes (the set of uncontrollable nodes is denoted by $\mathcal{U}$). The network operator can only control the routing behavior for controllable nodes while the routing actions taken by uncontrollable nodes cannot be regulated and are only observable at the end of each time slot. In this case, the network routing vector $\mathbf{f}(t)$ can be decomposed into two parts: $\mathbf{f}(t)=(\mathbf{f}^{c}(t),\mathbf{f}^{u}(t))$. Here, $\mathbf{f}^{c}(t)=\{f_{ijk}(t)\}_{i\in\mathcal{C}}$ represents the routing decisions made by controllable nodes (referred to as the controllable action) and $\mathbf{f}^{u}(t)=\{f_{ijk}(t)\}_{i\in\mathcal{U}}$ corresponds to the routing decisions made by uncontrollable nodes (referred to as the uncontrollable action). The routing vectors $\mathbf{f}^{c}(t)$ and $\mathbf{f}^{u}(t)$ are constrained within some action spaces $\mathcal{F}^{c}_{\omega_{t}}$ and $\mathcal{F}^{u}_{\omega_{t}}$, respectively, that may depend on the current network event $\omega_{t}$, respectively. The action space for all nodes is denoted by $\mathcal{F}_{\omega_{t}}=\mathcal{F}_{\omega_{t}}^{c}\cup\mathcal{F}_{\omega_{t}}^{u}$. The action space can be used to specify routing constraints (e.g., the total transmission rate over each link should not exceed its capacity) or describe scheduling constraints (e.g., each node can only transmit to one of its neighbors in each time slot).

Note that when there is not enough backlog to transmit, the actual number of transmitted packets may be less than the offered transmission rate. In particular, we denote by $\widetilde{f}_{ijk}(Q_{ik}(t))$ (or simply $\widetilde{f}_{ijk}(t)$ if the context is clear) the actual number of transmitted packets in flow $k$ over link $i\rightarrow j$ in slot $t$ under the current queue length $Q_{ik}(t)$. Clearly, we have $\widetilde{f}_{ijk}(Q_{ik}(t))\leq\min\{f_{ijk}(t),Q_{ik}(t)\}$. We further assume that the routing decision can always be chosen to respect the backlog constraints (but the actual actions may not necessarily be queue-respecting). This can be done simply by never attempting to transmit more data than we have. Under such notations, the queuing dynamics are given by

[TABLE]

where $[z]^{+}=\max\{z,0\}$. We also make the following boundedness assumption: the amount of exogenous arrivals and the offered transmission rate in each time slot are bounded by some constant $D$, i.e.,

[TABLE]

A network control policy $\pi$ is a mapping from the observed network event $\omega$ and queue length vector $\mathbf{Q}$ to a feasible routing action. In particular, denote by $\pi_{c}:(\omega,\mathbf{Q})\mapsto\mathbf{f}^{c}$ a controllable policy and $\pi_{u}:(\omega,\mathbf{Q})\mapsto\mathbf{f}^{u}$ an uncontrollable policy. In this paper, we assume that the uncontrollable policy $\pi_{u}$ remains fixed over time but is unknown to the network operator. Our objective is to find a controllable policy $\pi_{c}$ such that network stability can be achieved, as is defined as follows.

Definition 1**.**

A network is rate stable if

[TABLE]

Rate stability means that the average arrival rate to each queue equals the average departure rate from that queue.

III Queue-Agnostic Uncontrollable Policy

In this section, we consider the scenario where the uncontrollable policy is queue-agnostic, which simply observes the current network event $\omega_{t}$ and makes a routing decision $\mathbf{f}^{u}(t)\in\mathcal{F}_{\omega_{t}}^{u}$ as a stationary function only of $\omega_{t}$, i.e., $\pi_{u}:\omega_{t}\mapsto\mathbf{f}^{u}(t)$. In the stochastic network optimization literature, such a policy is also referred to as an $\omega$-only policy [11]. Despite their simple form, $\omega$-only policies can capture a wide range of network control protocols in practice, such as shortest-path routing protocols (e.g., OSPF, RIP) , multi-path routing protocols (e.g., ECMP) and randomized routing protocols.

Unfortunately, even under simple $\omega$-only uncontrollable policies, existing routing algorithms may fail to stabilize the network. For example, as is illustrated in Figure 2, the well-known Backpressure routing algorithm achieves low throughput when uncontrollable node uses queue-agnostic policies. In this example, the failure is due to the fact that some uncontrollable node uses a non-stabilizing policy that does not preserve flow conservation but the Backpressure algorithm is not aware of this non-stabilizing behavior.

In this section, we propose a low-complexity algorithm that learns the behavior of uncontrollable nodes and achieves network stability under any $\omega$-only uncontrollable policy.

III-A Tracking-MaxWeight Algorithm

Now we introduce an algorithm that achieves network stability whenever uncontrollable nodes use an $\omega$-only policy. The algorithm is called Tracking-MaxWeight (TMW), which enhances the original MaxWeight algorithm [26] with an explicit learning of the policy used by uncontrollable nodes. Throughout this section, we let $\{\mathbf{f}^{u}(t)\}_{t\geq 0}$ be the sequence of routing actions that are actually executed by uncontrollable nodes.

The details of the TMW algorithm are presented in Algorithm 1. In each slot $t$, the TMW algorithm generates the routing actions $\mathbf{g}^{c}(t)=\{g_{ijk}(t)\}_{i\in\mathcal{C}}$ for controllable nodes and also produces an “imagined” routing action $\mathbf{g}^{u}(t)=\{g_{ijk}(t)\}_{i\in\mathcal{U}}$ for uncontrollable nodes, by solving the optimization problem (2). With these calculated actions, the TMW algorithm then updates two virtual queues. The first virtual queue $\mathbf{X}(t)$ tries to emulate the physical queue $\mathbf{Q}(t)$ but assumes that the imagined uncontrollable action $\mathbf{g}^{u}(t)$ is applied (while the physical queue is updated using the true uncontrollable action $\mathbf{f}^{u}(t)$). The second virtual queue $\mathbf{Y}(t)$ tracks the cumulative difference between the imagined uncontrollable actions $\{\mathbf{g}^{u}(t)\}_{t\geq 0}$ and the actual uncontrollable actions $\{\mathbf{f}^{u}(t)\}_{t\geq 0}$. In particular, we use $\Delta_{ijk}(t)$ to measure the difference between the imagined routing action $g_{ijk}(t)$ and the true routing action $f_{ijk}(t)$ taken by uncontrollable node $i\in\mathcal{U}$, which is given by

[TABLE]

where $\widetilde{f}_{ijk}(t)$ is the actual number of transmitted packets under the true routing action $f_{ijk}(t)$ given the current queue backlog $\mathbf{Q}(t)$. Note that for each controllable node $i\in\mathcal{C}$, we simply set $\Delta_{ijk}(t)=0$.

The optimization problem (2) aims at maximizing a weighted sum of flow variables, which is similar to the optimization problem solved in the original MaxWeight algorithm [26] except for the setting of weights. In the original MaxWeight algorithm, the weight is $W_{ijk}(t)=Q_{ik}(t)-Q_{jk}(t)$ corresponding to the physical queue backlog differential, while in the Tracking-MaxWeight algorithm the weight $W_{ijk}(t)=X_{ik}(t)-X_{jk}(t)-Y_{ijk}(t)$ accounts for both the backlog differential for virtual queue $\mathbf{X}(t)$ and the backlog of virtual queue $\mathbf{Y}(t)$. The derivation of (2) is based on the minimization of quadratic Lyapunov drift terms for the two virtual queues:

[TABLE]

where the first term corresponds to the Lyapunov drift of virtual queue $\mathbf{X}(t)$ and the second term is the Lyapunov drift of virtual queue $\mathbf{Y}(t)$. Note that the minimization is done over controllable actions $\mathbf{g}^{c}(t)$ and “imagined” uncontrollable actions $\mathbf{g}^{u}(t)$. Cleaning up irrelevant constants, i.e., $a_{ik}(t)$ and $\widetilde{f}_{ijk}(t)$, and rearranging terms yield the optimization problem (2).

Next we show that Tracking-MaxWeight achieves stability whenever uncontrollable nodes use an $\omega$-only policy and the network is within the stability region, i.e., there exists a sequence of feasible routing vectors $\{\mathbf{f}^{c}(t)\}_{t\geq 0}$ for controllable nodes such that

[TABLE]

where

[TABLE]

is the long-term average actual flow transmission rate under $\{\mathbf{f}^{c}(t),\mathbf{f}^{u}(t)\}_{t\geq 0}$ and $\{\mathbf{Q}^{*}(t)\}_{t\geq 0}$ is the corresponding optimal queue length trajectory. In other words, (3) requires that flow conservation should be preserved for every queue under the optimal controllable policy, otherwise no algorithm can stabilize the network. It is important to note that in (3) the flow conservation law is with respect to the actual transmissions since an uncontrollable node may not preserve flow conservation in terms of its offered transmissions (e.g., in Figure 2, the offered incoming rate to node 3 is 40 while the offered outgoing rate from node 3 is 0). The only way to stabilize these nodes is by limiting the amount of backlog such that the actual endogenous arrivals to these nodes are smaller. The performance of Tracking-MaxWeight is given in the following theorem.

Theorem 1**.**

When uncontrollable nodes use an $\omega$-only policy and the network is within the stability region, Tracking-MaxWeight achieves rate stability.

Proof.

The proof first shows that the two virtual queues $\mathbf{X}(t)$ and $\mathbf{Y}(t)$ can be stabilized by the TMW algorithm by using the Lyapunov drift analysis. Then we prove that whenever the two virtual queues are stable, the physical queue $\mathbf{Q}(t)$ is also stable. See Appendix A for details. ∎

IV Queue-Dependent Uncontrollable Policy

The previous section investigated the scenario where uncontrollable nodes use a queue-agnostic policy (i.e., $\omega$-only policy). In this section, we study a more general case where the uncontrollable policy may be queue-dependent, which can be used to describe many state-of-the-art optimal network control protocols. For example, the well-known Backpressure algorithm makes routing decisions based on the currently observed queue length vector. In this scenario, the uncontrollable policy is a fixed mapping from the observed network event $\omega_{t}$ and the observed queue length vector $\mathbf{Q}(t)$ to a routing vector $\mathbf{f}^{u}(t)$ for uncontrollable nodes, i.e., \pi_{u}:\Big{(}\omega_{t},\mathbf{Q}(t)\Big{)}\mapsto\mathbf{f}^{u}(t).

Note that the queueing dynamics are

[TABLE]

Since for each $j\in\mathcal{U}$, its routing variable $f_{jik}(t)$ is an arbitrary (unknown) function of $\mathbf{Q}(t)$, the above queueing dynamics could depend on $\mathbf{Q}(t)$ in an arbitrary (unknown) way that is not in the simple piecewise-linear form as in the classic Lindley recursion. As a result, we rewrite the queueing dynamics as

[TABLE]

where $\beta(\cdot)$ is some unknown function that depends on our controllable routing action $\mathbf{f}^{c}(t)$, the current queue length vector $\mathbf{Q}(t)$ and the observed network event $\omega_{t}$.

Due to the unknown queueing dynamics, many analytical tools for optimal network control break down. For example, the previous Tracking-MaxWeight algorithm utilizes the Lyapunov drift analysis which is not applicable if the queueing dynamics do not follow the Lindley recursion. As a result, optimal network control becomes very challenging and fundamentally different from the traditional stochastic network optimization framework. In the following, we first formulate the problem a Markov Decision Process (MDP) with unknown dynamics and then propose a new reinforcement learning algorithm that can achieve network stability under mild conditions.

Before moving on to the technical details, we first introduce some notations and assumptions that will be used throughout this section. For convenience, we define action $\alpha_{t}\triangleq\mathbf{f}^{c}(t)$ and simply write “controllable routing action $\mathbf{f}^{c}(t)$” as “action $\alpha_{t}$”, since the uncontrollable routing action $\mathbf{f}^{u}(t)$ has been implicitly treated as a part of the environment (see queueing dynamics (4)). For the same reason, “controllable policy $\pi_{c}$” and “policy $\pi$” are also used interchangeably. The action space for $\alpha_{t}$ is denoted by $\mathcal{A}$ which is assumed to be fixed and finite. We also make the following assumption regarding the optimal system performance.

Assumption 1**.**

There exists a policy $\pi^{*}$ such that $\sum_{i,k}Q^{*}_{ik}(t)<\infty$ with probability 1 for any $t\geq 0$, where $\mathbf{Q}^{*}(t)$ is the queue length vector in slot $t$ under policy $\pi^{*}$.

In other words, it is required that the total queue length should remain bounded under an optimal policy $\pi^{*}$ otherwise there is no hope for stabilizing the network. In essence, Assumption 1 requires that the network be stabilizable by some controllable policy $\pi^{*}$.

IV-A MDP Formulation

We formulate the problem of achieving network stability as an MDP $M=(\mathcal{A},\mathcal{S},\theta,P)$. Here $\mathcal{A}$ is the routing action space for controllable nodes, and $\mathcal{S}$ is the state space that corresponds to the queue length vector space $\mathcal{Q}$. The cost function $\theta(\alpha_{t},\mathbf{Q}(t))$ under action $\alpha_{t}$ and state $\mathbf{Q}(t)$ is given by $\theta(\alpha_{t},\mathbf{Q}(t))=\sum_{i,k}Q_{ik}(t)$, which corresponds to the sum of queue lengths in slot $t$. In addition, $P$ is the state transition matrix, where $P(\mathbf{Q}^{\prime}|\mathbf{Q},\alpha)$ is the probability that the next state is $\mathbf{Q}^{\prime}$ when action $\alpha$ is taken under the current state $\mathbf{Q}$. Note that the transition matrix $P$ is generated according to the queueing dynamics (4), and that the influence of network event and uncontrollable routing action has been implicitly incorporated into the probabilistic transition matrix $P$. Note also that the queueing dynamics $\beta(\cdot)$ are unknown, so this is an MDP with unknown dynamics, which is also referred to as a Reinforcement Learning (RL) problem [23].

Let J^{\pi}\Big{(}M,\mathbf{Q}(0)\Big{)} be the time-average expected total queue length when policy $\pi$ is applied in MDP $M$ and the initial queue length vector is $\mathbf{Q}(0)$, i.e.,

[TABLE]

where the expectation $\mathbf{E}_{\pi,M}[\cdot]$ is with respect to the randomness of the queue length trajectory $\{\mathbf{Q}(t)\}_{t\geq 0}$ when policy $\pi$ is applied in MDP $M$. Also let J^{*}\Big{(}M,\mathbf{Q}(0)\Big{)}=\min_{\pi}J^{\pi}\Big{(}M,\mathbf{Q}(0)\Big{)} be the minimum time-average expected queue length under an optimal policy $\pi^{*}$. Our objective is to find an optimal policy that solves the MDP and achieves the minimum average queue length.

IV-B Challenges to Solving the MDP

The MDP has an unknown transition structure, which gives rise to an “exploration-exploitation” tradeoff. On one hand, we need to exploit the existing knowledge to make the best (myopic) decision; on the other hand, it is necessary to explore new states in order to learn which states may lead to lower costs in the future. Moreover, there might be some “trapping” sub-optimal states that take a long time (or is even impossible) for any policy to escape. Any algorithm that has zero knowledge about system dynamics at the beginning is likely to get trapped in these states during the exploration phase. Therefore, we need to impose restrictions on the transition structure in the MDP model. In particular, we restrict our consideration to weakly communicating MDPs with finite communication time, defined as follows.

Assumption 2**.**

For any two queue length vectors $\mathbf{Q}$ and $\mathbf{Q}^{\prime}$ (except for those which are transient under every policy), there exists a policy $\pi$ that can move from $\mathbf{Q}$ to $\mathbf{Q}^{\prime}$ within $L||\mathbf{Q}^{\prime}-\mathbf{Q}||_{1}$ time slots (in expectation), where $L$ is a constant.

In other words, it is assumed that there is no “trapping” state in the system otherwise no reinforcement learning algorithm can be guranteed to avoid the traps and optimally solve the MDP. Note that in a weakly communicating MDP, the optimal average cost does not depend on the initial state (cf. [18], Section 8.3.3). Thus we drop the dependence on the initial state $\mathbf{Q}(0)$, and write the optimal average cost (queue length) as $J^{*}(M)$.

Another challenge is that the MDP has a countably-infinite state space (i.e., queue length vector space). Existing reinforcement learning methods that can handle such an infinite state space are mostly heuristic-based (e.g., [8][5][20]), and do not have any performance guarantees. On the other hand, there are a few reinforcement learning algorithms that do have good performance guarantees, but these algorithms require that the size of the state space be relatively small. Even if we consider a finite time horizon $T$, the size of the queue length vector space could be up to $O(T^{N})$ (assuming bounded arrivals in each slot), which could lead to weak performance bounds. For example, in the UCRL algorithm [1, 2], the regret bound is $O(S\sqrt{T})$, where $S$ is the size of the state space. If UCRL is applied in our context, the resulting regret bound would be $O(T^{N+0.5})$ which is a trivial super-linear regret bound.

IV-C TUCRL Algorithm

In this section, we develop an algorithm that achieves network stability under Assumptions 1 and 2. We call our algorithm Truncated Upper Confidence Reinforcement Learning (TUCRL), as it combines the model-based UCRL algorithm [1, 2] with a queue truncation technique that resolves the infinite state space problem.

Specifically, consider a truncated system where new exogenous packet arrivals are dropped when the total queue length $\sum_{i,k}Q_{ik}(t)$ reaches $V-1$ for some threshold $V\geq 1$. In such a truncated system, the state space is the truncated queue length vector space $\mathcal{Q}_{V}$ which contains all queue length vectors where the length of each queue does not exceed $V-1$. In order for packet dropping to be feasible, we assume that there is an admission control action that can shed new exogenous packets as needed.

Our TUCRL algorithm applies the model-based UCRL algorithm [1, 2] in the truncated system, which maintains an estimation for the unknown queueing dynamics and then computes the optimal policy under the estimated dynamics. It applies the “optimistic principle” for exploration, where under-explored state-action pairs are assumed to be able to result in lower costs, which implicitly encourages the exploration of novel state-action pairs.

The detailed description of TUCRL is presented in Algorithm 2, which is similar to the standard UCRL algorithm except that queue truncation is applied when appropriate. Specifically, the TUCRL algorithm proceeds in episodes, and the length of each episode is dynamically determined. In episode $\ell$, the TUCRL algorithm first constructs an empirical estimation $\hat{P}$ for the transition matrix based on historical observations (step 1). In particular, the estimated transition probability from state $\mathbf{Q}$ to $\mathbf{Q}^{\prime}$ under action $\alpha$ is

[TABLE]

where $n_{\ell}(\mathbf{Q},\alpha)$ is the cumulative number of visits to state-action pair $(\mathbf{Q},\alpha)$ up until the beginning of episode $\ell$ and $n_{\ell}(\mathbf{Q},\alpha,\mathbf{Q}^{\prime})$ is the number of times that transition $(\mathbf{Q},\alpha)\rightarrow\mathbf{Q}^{\prime}$ happens up to the beginning of episode $\ell$. Note that if $n_{\ell}(\mathbf{Q},\alpha)=0$, the estimated transition probability is set to be zero.

Then the TUCRL algorithm constructs an upper confidence set $\mathcal{M}_{\ell}$ for all plausible MDP models based on the empirical estimation $\hat{P}$ (step 2). The upper confidence set is constructed in a way such that it contains the true MDP model with high probability. Specifically, the upper confidence set $\mathcal{M}_{\ell}$ contains all the MDPs with truncated queue length space $\mathcal{Q}_{V}$ and transition matrix $P\in\mathcal{P}_{\ell}$ where

[TABLE]

Here, $V$ is the queue truncation threshold, $t_{\ell}$ is the starting time of episode $\ell$ and $C\triangleq 2(2ND+1)^{N}$ is a constant.

Next, the TUCRL algorithm selects an “optimistic MDP” $M_{\ell}$ that yields the minimum average queue length among all the plausible MDPs in the confidence set $\mathcal{M}_{\ell}$, and computes a nearly-optimal policy $\pi_{\ell}$ under MDP $M_{\ell}$ (step 3). The joint selection of the optimistic MDP and the calculation of the nearly-optimal policy are referred to an optimistic planning [9]. There are many efficient methods for performing optimistic planning, such as Extended Value Iteration [2] and OP-MDP [24]. For completeness, we provide a description of Extended Value Iteration in Appendix B.

Finally, the computed policy $\pi_{\ell}$ is executed until the stopping condition of episode $\ell$ is triggered. An episode ends when the number of visits to some state-action pair doubles, i.e., when we encounter a state-action pair $(\mathbf{Q}(t),\alpha_{t})$ such that its visiting frequency in episode $\ell$ ($v_{\ell}(\mathbf{Q}(t),\alpha_{t})$) equals its cumulative visiting frequency up to the beginning of episode $\ell$ ($n_{\ell}(\mathbf{Q}(t),\alpha_{t})$). We will show that this stopping condition guarantees that the total number of episodes up to time $T$ is $O(V^{N}\log T)$ (see Appendix D). Note that during the execution of policy $\pi_{\ell}$, new packet arrivals may be dropped if the total queue length exceeds $V-1$. Here, the dropped packets could be any new arrivals to any queue. We will prove that the fraction of dropped packets is negligible if the threshold $V$ is properly selected.

IV-D Performance of TUCRL Algorithm

The following theorem characterizes the performance of the TUCRL algorithm regarding its queue length, packet dropping rate and convergence rate.

Theorem 2**.**

Under Assumptions 1 and 2, the performance of the TUCRL algorithm is as follows.

$\bullet$* (Queue Length) The time-average expected queue length converges to a bounded value:*

[TABLE]

$\bullet$* (Packet Dropping Rate) The long-term expected fraction of dropped packets is*

[TABLE]

where $\eta_{T}$ is the fraction of dropped packets within $T$ slots.

$\bullet$* (Convergence Rate) The time-average expected queue length after $T$ slots is within a \widetilde{\mathcal{O}}\Big{(}\frac{poly(V)}{\sqrt{T}}\Big{)}-neighborhood of the steady-state expected queue length, where $poly(V)$ is some polynomial in $V$ and $\widetilde{\mathcal{O}}$ is the big-O notation that ignores any logarithmic term.*

Proof.

We first find an upper bound on the total queue length under TUCRL in the truncated system. Based on the queue length upper bound, we further analyze the fraction of time when queue truncation is triggered by using concentration inequalities. See Appendix C for details. ∎

There are several important observations regarding Theorem 2. First, the TUCRL algorithm achieves bounded queue length by dropping a negligible fraction of packets under a suitably large value of $V$. Second, there is a three-way tradeoff between total queue length (delay), packet dropping rate (throughput) and convergence rate. For example, by increasing the value of $V$, the packet dropping rate becomes smaller (i.e., throughput becomes higher) but the total queue length (delay) increases and the convergence becomes slower. Similar three-way tradeoffs between utility, delay and convergence rate are discussed in [7].

Complexity of TUCRL. The time complexity of TUCRL is dominated by the complexity of the optimistic planning module (step 3) which is implementation-dependent. For example, if a naive Extended Value Iteration (see Appendix B) is used, the time complexity of each value iteration step is exponential in the number of queues and thus cannot scale to large-scale problems. One way to scale the optimisic planning module is by using approximate dynamic programming that employs various approximation techniques in the planning procedure, such as using linear functions or neural networks to approximate the value function (see [17] for a comprehensive introduction). Recent deep reinforcement learning techniques may also be leveraged to efficiently perform value iterations in large-scale problems, such as Randomized Least-Squares Value Iteration (RLSVI) [14], Value Iteration Networks (VIN) [25] and Value Prediction Networks (VPN) [12]. Such approximations will not lead to significant changes in the performance of TUCRL since we only require an approximate solution in step 3.

V Simulation Results

V-A Scenario 1: Queue-Agnostic Uncontrollable Policy

We first study the partially-controllable network shown in Figure 3. There are two flows: $1\rightarrow 4$ and $6\rightarrow 4$. Each node in the network needs to make a routing and scheduling decision in every time slot. The constraint is that each node can transmit to only one of its neighbors in each time slot and the transmission rate over each link cannot exceed its capacity. Node 2 and node 3 are uncontrollable nodes that use randomized queue-agnostic policies. Specifically, uncontrollable node 2 uses a randomized routing algorithm that transmits any packets it received to either node 3 or node 5 with an equal probability in each time slot. Uncontrollable node 3 uses a randomized scheduling policy that serves flow $1\rightarrow 4$ or flow $6\rightarrow 4$ with an equal probability in each time slot. The arrival rate of flow $6\rightarrow 4$ is 5. In this case, it can be shown that the maximum supportable arrival rate for flow $1\rightarrow 4$ is 25 given the routing constraints and the behavior of uncontrollable nodes.

We have shown in Section III that the Tracking-MaxWeight (TMW) algorithm achieves the optimal throughput in this scenario. In Figure 4(a), we compare Tracking-MaxWeight with the well-known MaxWeight algorithm (i.e., BackPressure routing), in terms of the supportable rate for flow $1\rightarrow 4$. Specifically, Figure 4(a) shows the total queue length achieved by MaxWeight and Tracking-MaxWeight under different system loads (if the load is $\rho$, then the arrival rate of flow $1\rightarrow 4$ is $25\rho$ while the arrival rate of flow $6\rightarrow 4$ is fixed to 5). It is observed that MaxWeight can only support around 40% arrivals (the queue length under MaxWeight blows up at load $\approx 0.4$). By comparison, our Tracking-MaxWeight achieves the optimal throughput.

We further examine the behavior of the Tracking-MaxWeight algorithm in Figure 4(b) and Figure 4(c). Specifically, Figure 4(b) shows the queue length trajectory for the physical queue $\mathbf{Q}(t)$ and the two virtual queues $\mathbf{X}(t),\mathbf{Y}(t)$. As our theory predicts, both the physical queue $\mathbf{Q}(t)$ and the two virtual queues $\mathbf{X}(t),\mathbf{Y}(t)$ are stable under the TMW algorithm. Figure 4(c) shows the learning curve of the TMW algorithm for the uncontrollable policy used by node $3$. In particular, node $3$ uses randomized scheduling that serves flow $1\rightarrow 4$ and flow $6\rightarrow 4$ with an equal probability 0.5. It is observed in Figure 4(c) that the TMW algorithm quickly learns the service probability for flow $1\rightarrow 4$ at node 3 (i.e., the “imagined uncontrollable action” in TMW approaches the true uncontrollable action).

V-B Scenario 2: Queue-Dependent Uncontrollable Policy

Next we study a more challenging scenario where the action taken by uncontrollable nodes is queue-dependent. In particular, consider the network topology shown in Figure 5 where node 2 and node 3 are uncontrollable. There is only one flow $1\rightarrow 4$ and the constraint is that each node can transmit to only one of its neighbours in each time slot. The policy used by the two uncontrollable nodes is as follows. Let $\mu_{24}(t)$ and $\mu_{34}(t)$ be the transmission rate that node 2 and node 3 allocates to the flow in slot $t$, respectively. Then

[TABLE]

As a result, the maximum throughput of 1 can be supported only if $Q_{2}(t)$ is small ($Q_{2}(t)\leq 10$) and $Q_{3}(t)$ is large ($Q_{3}(t)>10$). Although this is an artificial example, it sheds light on the challenges when uncontrollable nodes use queue-dependent policies: any throughput-optimal algorithm should be able to efficiently learn which queue length region can support the maximum throughput and keep the queue length within this region.

We first compare the throughput performance of TUCRL with MaxWeight and Tracking-MaxWeight. Note that the TUCRL algorithm occasionally drops packets. In order to make a fair comparison, the throughput performance is measured with respect to the number of packets that have been delivered. It is observed in Figure 8 that TUCRL achieves the optimal throughput while MaxWeight or Tracking-MaxWeight only deliver a throughput of 0.5 in this scenario. It should be noted that the TUCRL algorithm takes longer time to learn and converge than MaxWeight or Tracking-MaxWeight.

Next we investigate the performance of the TUCRL algorithm under different values of the truncation threshold $V$. As we proved in Theorem 2, the value of $V$ determines a three-way tradeoff between queue length, packet dropping rate and convergence rate. As is illustrated in Figure 8 and Figure 8, a larger value of $V$ leads to a larger queue length and the convergence becomes slower, but the fraction of dropped packets becomes smaller. Note that when $V=20$ and $V=30$, the fraction of dropped packets becomes very small as time goes by. In contrast, when $V=5$, the fraction of dropped packets remains non-negligible ($\sim$ 60%) since the TUCRL algorithm cannot explore the “throughput-optimal region” where $Q_{2}(t)\leq 10\text{ and }Q_{3}(t)>10$ with a queue truncation threshold $V=5$.

VI Conclusions

In this paper, we study optimal network control algorithms that stabilize a partially-controllable network where a subset of nodes are uncontrollable. We first study the scenario where the uncontrollable nodes use a queue-agnostic policy and propose a simple throughput-optimal Tracking-MaxWeight algorithm that enhances the original MaxWeight algorithm with an explicit learning of uncontrollable behavior. Then we investigate the scenario where the uncontrollable policy may be queue-dependent. This problem is formulated as an MDP and we develop a reinforcement learning algorithm called TUCRL that achieves a three-way tradeoff between throughput, delay and convergence rate.

Appendix A Proof to Theorem 1

We first introduce a lemma that characterizes the load of the two virtual queues $\mathbf{X}(t)$ and $\mathbf{Y}(t)$.

Lemma 1**.**

There exists an $\omega$-only controllable policy $\widehat{\pi}_{c}:\omega_{t}\mapsto\bm{\mu}^{c}(t)$ and an $\omega$-only uncontrollable policy $\widehat{\pi}_{u}:\omega_{t}\mapsto\bm{\mu}^{u}(t)$ such that

[TABLE]

[TABLE]

where $\mu_{ijk}=\mathbb{E}[\mu_{ijk}(t)]$ is the expected flow transmission rate under the $\omega$-only policy $\widehat{\pi}_{c}$ or $\widehat{\pi}_{u}$.

Proof.

By our assumption, there exists a sequence of queue-respecting routing vectors $\{\mathbf{r}(t)\}_{t\geq 0}$ such that

[TABLE]

By Lemma 2 in [10], there exists an $\omega$-only controllable policy $\widehat{\pi}_{c}:\omega_{t}\mapsto\bm{\mu}^{c}(t)$ and an $\omega$-only uncontrollable policy $\widehat{\pi}_{u}:\omega_{t}\mapsto\bm{\mu}^{u}(t)$ such that

[TABLE]

As a result, we have

[TABLE]

By (3) we have

[TABLE]

which completes the proof. ∎

Next we prove that the two virtual queues can be stabilized by the TMW algorithm.

Lemma 2**.**

Under the TMW algorithm, we have

[TABLE]

[TABLE]

Proof.

Define the Lyapunov function as

[TABLE]

By the evolution of virtual queue $\mathbf{X}(t)$, we have

[TABLE]

where the inequality is due to the boundedness assumptions that $a_{ik}(t)\leq D$ and $g_{ijk}(t)\leq D$ for any $i,j,k$. Similarly, for virtual queue $\mathbf{Y}(t)$ we have

[TABLE]

where we rewrite $\Delta_{ijk}(t)=g_{ijk}(t)-\widetilde{f}_{ijk}(Q_{ik}(t))$ as \Delta_{ijk}\big{(}g_{ijk}(t),f_{ijk}(t),Q_{ik}(t)\big{)} to explicitly emphasize on the dependence of $\Delta_{ijk}(t)$ on the imagined routing action $g_{ijk}(t)$, the true routing action $f_{ijk}(t)$ and the current queue length $Q_{ik}(t)$.

As a result, the conditional expected Lyapunov drift $\delta(t)\triangleq\mathbb{E}[\Phi(t+1)-\Phi(t)|\mathbf{X}(t),\mathbf{Y}(t)]$ can be bounded by

[TABLE]

where the second inequality is due to the operation of TMW and $\bm{\mu}(t)$ is given in Lemma 1 which shows that

[TABLE]

In addition, for any $i,j,k$, when there are enough queue backlogs ($Q_{ik}(t)\geq ND$) we have

[TABLE]

where $\mathbf{Q}^{*}(t)$ is the queue length vector under the optimal controllable policy and the last equality is due to Lemma 1. As a result, when there are enough backlogs (i.e., $Q_{ik}(t)\geq ND$ for any $i,k$) we have

[TABLE]

Let $t_{0}$ be the last time when there exists a queue $(i,k)$ such that $Q_{ik}(t)<ND$, i.e., $Q_{ik}\geq ND$ for any $t\geq t_{0}$ and any $i,k$. Without loss of generality, we assume $t_{0}=0$. Summing over $t=0,\cdots,T-1$ and using law of iterated expectation, we have

[TABLE]

It follows that

[TABLE]

which implies that

[TABLE]

Assuming that $\sum_{i,k}X_{ik}(0)<\infty$ and $\sum_{i,j,k}Y_{ijk}(0)<\infty$, we have

[TABLE]

Similarly, we have

[TABLE]

This completes the proof. ∎

Finally, we show that as long as the two virtual queues $\mathbf{X}(t)$ and $\mathbf{Y}(t)$ are stable, then the physical queue $\mathcal{Q}(t)$ is also stable.

Lemma 3**.**

For any $i,k$ and $t\geq 0$, we have $Q_{ik}(t)\leq X_{ik}(t)+\sum_{j\in\mathcal{N}}Y_{ijk}(t)-\sum_{j\in\mathcal{U}}Y_{jik}(t)$.

Proof.

We prove this lemma by induction on $t$. The base case trivially holds true since we initialize $X_{ik}(0)=Q_{ik}(0)$ and $Y_{ijk}(0)=0$. Now suppose that $Q_{ik}(t)\leq X_{ik}(t)+\sum_{j\in\mathcal{N}}Y_{ijk}(t)-\sum_{j\in\mathcal{U}}Y_{jik}(t)$ holds true for some $t\geq 0$. Then for any $i\in\mathcal{U}$ we have

[TABLE]

where the first inequality is due to the induction and simple algebra. Similar induction applies to every controllable node $i\in\mathcal{C}$. This completes the induction proof. ∎

By Lemma 2 and Lemma 3, we can finally conclude that

[TABLE]

which completes the proof to Theorem 1.

Appendix B Extended Value Iteration

In this appendix, we introduce Extended Value Iteration (EVI) [2] as one of the approaches for optimistic planning (see step 3 in Algorithm 2). Extended Value Iteration is similar to the canonical Value Iteration but applies to MDPs with “extended action space” where the additional action is the selection of the optimistic MDP (more precisely, the selection of the optimistic transition matrix) that yields the minimum average cost among a set of plausible MDPs. In particular, let $w_{j}(\mathbf{Q})$ be the value function for state $\mathbf{Q}$ obtained after iteration $j$. Then Extended Value Iteration proceeds as follows. For any $\mathbf{Q}\in\mathcal{Q}_{V}$

[TABLE]

where $\mathcal{P}_{\ell}(\mathbf{Q},\alpha)$ is given in (6). The inner minimization problem can be easily solved using the following greedy algorithm which tries to push as much probability mass to the state with the largest value.

Greedy Algorithm: find optimistic transition probability $p(\cdot)$ for a particular state-action pair $(\mathbf{Q},\alpha)$

Input: Estimated transition probability $\hat{P}(\cdot|\mathbf{Q},\alpha)$ and distance $d(\mathbf{Q},\alpha)$ as given in (6)

1. Sort states in $\mathcal{Q}_{V}$ in descending order according to their $w_{j}(\cdot)$ value. We assume that the sorted state space is denoted by $\mathcal{Q}^{\prime}_{V}=\{\mathbf{Q}_{1}^{\prime},\mathbf{Q}_{2}^{\prime},\cdots,\mathbf{Q}_{n}^{\prime}\}$ where

[TABLE]

2. Set

[TABLE]

[TABLE]

3. Set $m=n$

4. While $\sum_{\mathbf{Q}^{\prime}\in\mathcal{Q}^{\prime}_{V}}p(\mathbf{Q}^{\prime})>1$ do

(a) Reset $p(\mathbf{Q}^{\prime}_{m})=\max\{0,1-\sum_{\mathbf{Q}^{\prime}\neq\mathbf{Q}^{\prime}_{m}}p(\mathbf{Q}^{\prime})\}$.

(b) Set $m=m-1$.

The distance $d(\mathbf{Q},\alpha)$ in the input is the range used in our confidence set construction (6), i.e.,

[TABLE]

The iteration stops when the following convergence criterion is triggered:

[TABLE]

which requires that the span of the average cost across all states be less than or equal to $\frac{1}{\sqrt{t_{\ell}}}$.

Appendix C Proof to Theorem 2

We first introduce some notations.

• •

Let $M(P)$ be an MDP model for the original untruncated system (where no packet dropping happens) with transition matrix $P$. Denote by $M^{*}\triangleq M(P^{*})$ the true MDP model for the original untruncated system with the true transition matrix $P^{*}$. Note that in an untruncated system, the state space is the countably-infinite queue length space $\mathcal{Q}$.

• •

Let $M_{V}(P)$ be a truncated MDP model with transition matrix $P$ for a truncated system with threshold $V$. Note that the state space in a truncated MDP model is the truncated queue length space $\mathcal{Q}_{V}$. Denote by $M^{*}_{V}$ the truncated true MDP model that is obtained by applying packet dropping to the original untruncated true MDP model $M^{*}$.

• •

Let $J^{*}$ be the optimal time-average expected queue length in the untruncated true MDP $M^{*}$. Similarly, denote by $J_{V}^{*}$ the optimal time-average expected queue length in the truncated true MDP $M_{V}^{*}$.

The following theorem provides an upper bound on the total queue length in a truncated system with threshold $V$.

Theorem 3**.**

With probability at least 1-O\Big{(}\frac{1}{T}+\frac{1}{V}\Big{)}, the TUCRL algorithm achieves

[TABLE]

where

[TABLE]

Proof.

See Appendix E. ∎

Next, we provide an upper bound on the fraction of dropped packets based on Theorem 3.

Theorem 4**.**

When $T$ is sufficiently large, with probability at least 1-O\Big{(}\frac{1}{T}+\frac{1}{V}\Big{)}, the fraction of dropped packets is

[TABLE]

Proof.

See Appendix J. ∎

Note that the above results depend on the optimal average queue length $J_{V}^{*}$ in the truncated system. The following lemma shows the relationship between $J_{V}^{*}$ and $J^{*}$ (the optimal average queue length in the original untruncated system).

Lemma 4**.**

\mathbb{P}[J_{V}^{*}=J^{*}]\geq 1-O\Big{(}\frac{1}{V}\Big{)}.

Proof.

See Appendix K. ∎

Combining Lemma 4, Theorem 3 and Theorem 4, we have that with probability at least 1-\Theta\Big{(}\frac{1}{T}+\frac{1}{V}\Big{)} the TUCRL algorithm achieves

[TABLE]

and

[TABLE]

On the other hand, the worst-case average queue length is at most $V$ and the worst-case packet dropping rate is 1, which happens with probability at most \Theta\Big{(}\frac{1}{T}+\frac{1}{V}\Big{)}. As a result, the expected average queue length under TUCRL is

[TABLE]

and the expected fraction of dropped packets is

[TABLE]

For any given $V<\infty$, taking $T\rightarrow\infty$ and noting that $J^{*}<\infty$ (see Assumption 1) yield the desirable performance bounds in Theorem 2.

Appendix D Number of Episodes in TUCRL

The following lemma provides an upper bound on the number of episodes in TUCRL.

Lemma 5**.**

The total number of episodes in TUCRL up to time $T\geq 1$ is bounded by

[TABLE]

Proof.

Let $\mathcal{L}(\mathbf{Q},\alpha)$ be the set of episodes where the number of visits to a state-action pair $(\mathbf{Q},\alpha)$ is doubled up to the beginning of time $T$, i.e.,

[TABLE]

where $L_{T}$ is the index of the episode that contains time slot $T$. Also let $\mathcal{K}=\{(\mathbf{Q},\alpha):n_{L_{T}}(\mathbf{Q},\alpha)\geq 1\}$ be the set of state-action pairs that have non-zero visits up to the beginning of episode $L_{T}$. Then we claim that |\mathcal{L}(\mathbf{Q},\alpha)|\leq\log_{2}\Big{(}n_{L_{T}}(\mathbf{Q},\alpha)\Big{)}+1 for any $(\mathbf{Q},\alpha)\in\mathcal{K}$. Indeed, if |\mathcal{L}(\mathbf{Q},\alpha)|>\log_{2}\Big{(}n_{L_{T}}(\mathbf{Q},\alpha)\Big{)}+1, for any $(\mathbf{Q},\alpha)\in\mathcal{K}$ we have

[TABLE]

which leads to a contradiction. As a result, we have

[TABLE]

where the second inequality is due to the concavity of $\log$ function. ∎

Appendix E Proof to Theorem 3

Define the regret (w.r.t. queue length) in episode $\ell$ as

[TABLE]

where $J_{V}^{*}$ is the optimal average queue length for the truncated true MDP $M_{V}^{*}$. Let $L_{T}$ be the number of episodes up to time $T$ (including the one that contains $T$). Without loss of generality, we assume that $t_{L_{T}+1}=T$ (i.e., $T-1$ is the end of episode $L_{T}$). Also denote by $\mathcal{L}_{1}=\{\ell\leq L_{T}:t_{\ell+1}<\log_{2}T\}$ the set of episodes whose ending time is less than $\log_{2}T$, and we similarly define $\mathcal{L}_{2}=\{\ell\leq L_{T}:t_{\ell+1}\geq\log_{2}T\}$. The regret up to time $T$ can be decomposed by

[TABLE]

where the inequality is due to the fact the total amount of exogenous arrivals (and thus the total queue length) within the first $\log_{2}T$ slots is at most $NKD\log_{2}T$ by our boundedness assumption. It turns our that the first term in (10) is of a lower order, and we focus on finding an upper bound on the second term. The following lemma shows that the constructed confidence set $\mathcal{M}_{\ell}$ contains the truncated true MDP model $M_{V}^{*}$ with high probability.

Lemma 6**.**

For any $T\geq 2$, the probability that the truncated true MDP $M_{V}^{*}$ belongs to the confidence set $\mathcal{M}_{\ell}$ for every episode $\ell\in\mathcal{L}_{2}$ is

[TABLE]

Proof.

See Appendix F. ∎

In the following, we only consider the case where $M_{V}^{*}\in\mathcal{M}_{\ell},~{}\forall\ell\in\mathcal{L}_{2}$ since this holds with high probability by Lemma 6. Let $M_{\ell}\triangleq M_{V}(P_{\ell})$ be the optimistic MDP selected in episode $\ell$ and define $J_{\ell}\triangleq J^{\pi_{\ell}}(M_{\ell})$ as the average queue length under MDP $M_{\ell}$ and the nearly-optimal policy $\pi_{\ell}$. Note that $J_{\ell}-J^{*}(M_{\ell})\leq\frac{1}{\sqrt{t_{\ell}}}$ by the accuracy requirement in Step 10 of the TUCRL algorithm. Moreover, due to the fact that the truncated true MDP $M_{V}^{*}\in\mathcal{M}_{\ell}$ and $M_{\ell}$ is the optimistic MDP that yields the minimum average queue length among all MDPs in $\mathcal{M}_{\ell}$, we have $J^{*}(M_{\ell})\leq J_{V}^{*}$ and thus

[TABLE]

Then we can decompose $\Delta_{\ell}$ as follows:

[TABLE]

Summing over all episodes $\ell\in\mathcal{L}_{2}$, we have

[TABLE]

We first find an upper bound for the second term in the RHS of (12) by using Lemma 19 in [2]: for any sequence of numbers $z_{1},\cdots,z_{n}$ with 0\leq z_{\ell}\leq Z_{\ell-1}:=\max\Big{\{}1,\sum_{i=1}^{\ell-1}z_{i}\Big{\}}, we have

[TABLE]

As a result, defining $z_{\ell}=t_{\ell+1}-t_{\ell}$ and Z_{\ell-1}=\max\Big{\{}1,t_{\ell}\Big{\}}, we have

[TABLE]

Then it remains to find an upper bound on the first term in the RHS of (12).

Lemma 7**.**

If $M_{V}^{*}\in\mathcal{M}_{\ell}$ for any $\ell\in\mathcal{L}_{2}$, then with probability at least $1-\frac{1}{T}$ we have

[TABLE]

Proof.

See Appendix G. ∎

Combining (10), (13), (12), Lemma 6 and Lemma 7, we can conclude that with probability at least 1-\Big{(}\frac{1}{T}+\frac{1}{V^{N+1}|\mathcal{A}|}+\frac{2}{V}\Big{)}

[TABLE]

which completes the proof to Theorem 3.

Appendix F Proof to Lemma 6

The $\ell_{1}$-deviation of the true distribution $p(\cdot)$ and the empirical distribution $\hat{p}(\cdot)$ over $m$ distinct events from $n$ samples is bounded according to Weissman et al. [27] by

[TABLE]

In our case, for each state-action pair $(\mathbf{Q},\alpha)$, the number of possible next states is at most $m=(2ND+1)^{N}$ since the increase/decrease in the length of each queue is at most $ND$ in every slot (including both exogenous and endogenous arrivals). Let $P_{V}^{*}$ be the transition probabilities for the truncated true MDP $M^{*}_{V}$. Then the number of samples we obtained for distribution $P_{V}^{*}(\cdot|\mathbf{Q},\alpha)$ up to the beginning of episode $\ell$ (i.e., when $t=t_{\ell}$) is $n_{\ell}(\mathbf{Q},\alpha)$. As a result, for episode $\ell$ and any state-action pair $(\mathbf{Q},\alpha)$ such that $n_{\ell}(\mathbf{Q},\alpha)\geq 1$, setting

[TABLE]

guarantees that

[TABLE]

Note that the above bound holds for every state-action pair $(\mathbf{Q},\alpha)$ such that $n_{\ell}(\mathbf{Q},\alpha)\geq 1$. If there have not been any observations for some state-action pair $(\mathbf{Q},\alpha)$, i.e., $n_{\ell}(\mathbf{Q},\alpha)=0$, the confidence intervals (6) trivially hold with probability 1. Thus, inequality (14) actually holds for every $\mathbf{Q}\in\mathcal{Q}_{V}$ and $\alpha\in\mathcal{A}$.

Applying the union bound over all possible state-action pairs in the truncated MDP, we have for each episode $\ell$

[TABLE]

i.e.,

[TABLE]

Let $\ell^{\prime}$ be the smallest episode index in $\mathcal{L}_{2}$. For any $\ell\in\mathcal{L}_{2}/\{\ell^{\prime}\}$, we have $t_{\ell}\geq\log_{2}T$. As a result, we have for any $T\geq 2$

[TABLE]

where the first inequality is derived by applying union bound to (15), the second inequality is due to the definition of $\mathcal{L}_{2}$, the third inequality is due to Lemma 5 which states that the total number of episodes up to time $T$ is at most $1+|\mathcal{Q}_{V}||\mathcal{A}|(\log_{2}T+1)$.

Appendix G Proof to Lemma 7

Note that $\sum_{t=t_{\ell}}^{t_{\ell+1}-1}\sum_{i,k}Q_{ik}(t)$ can be re-arranged according to the visiting frequency $v_{\ell}(\mathbf{Q},\alpha)$ for each state-action pair $(\mathbf{Q},\alpha)$ during episode $\ell$:

[TABLE]

As a result, we have

[TABLE]

We first provide an observation regarding the value functions we derived in Extended Value Iteration (EVI) (7).

Observation 1**.**

If the truncated true MDP $M_{V}^{*}$ belongs to the confidence set $\mathcal{M}_{\ell}$, then for any iteration $j$ in EVI, the range of the derived value function $w_{j}(\cdot)$ is bounded by $O(V^{2})$, i.e.,

[TABLE]

Proof.

Note that in the $j$th iteration of EVI, the derived value function $w_{j}(\mathbf{Q})$ is the sum of expected queue lengths within $j$ time steps under an optimal (possibly non-stationary) $j$-step policy starting in state $\mathbf{Q}$ in the MDP with extended action set (the extra action is to select an MDP within $\mathcal{M}_{\ell}$). If there are two states $\mathbf{Q},\mathbf{Q}^{\prime}\in\mathcal{Q}_{V}$ such that $w_{j}(\mathbf{Q})-w_{j}(\mathbf{Q}^{\prime})>LN^{2}V^{2}$, then an improved value for state $\mathbf{Q}$ could be achieved by adopting the following policy. First, follow a policy which always selects the truncated true MDP $M_{V}^{*}$ (this is a feasible action in the extended action set by our assumption) and moves from $\mathbf{Q}$ to $\mathbf{Q}^{\prime}$ most quickly in $M_{V}^{*}$, which takes at most $L||\mathbf{Q}^{\prime}-\mathbf{Q}||_{1}\leq LNV$ steps on average by Assumption 2. After reaching state $\mathbf{Q}^{\prime}$, we follow the optimal $j$-step policy for $\mathbf{Q}^{\prime}$ for the remaining $j-LNV$ time steps. Note that the sum of accumulative queue lengths before reaching state $\mathbf{Q}^{\prime}$ is at most $LN^{2}V^{2}$ since in each time step the total queue length is at most $NV$ in the truncated system. As a result, the constructed policy yields $w_{j}(\mathbf{Q})\leq w_{j}(\mathbf{Q}^{\prime})+LN^{2}V^{2}$, contradicting our assumption that $w_{j}(\mathbf{Q})-w_{j}(\mathbf{Q}^{\prime})>LN^{2}V^{2}$. ∎

It is a direct consequence of Theorem 8.5.6. of [18], that when the convergence criterion (8) holds at iteration $j$, then

[TABLE]

In the following, letter $j$ will be reserved to reference the iteration where the convergence criterion (8) is triggered. Expanding $w_{j+1}(\mathbf{Q})$ according to (7), we have

[TABLE]

and hence by (18) we have

[TABLE]

which implies that for any $\mathbf{Q}\in\mathcal{Q}_{V}$

[TABLE]

where the second equality holds because \sum_{\mathbf{Q}^{\prime}\in\mathcal{Q}_{V}}P_{\ell}\Big{(}\mathbf{Q}^{\prime}|\mathbf{Q},\pi_{\ell}(\mathbf{Q})\Big{)}=1.

Define $h_{\ell}(\mathbf{Q})\triangleq w_{j}(\mathbf{Q})-\min_{\mathbf{Q}\in\mathcal{Q}_{V}}w_{j}(\mathbf{Q})$ (we use the subscript $\ell$ to reference the episode). It is clear from Observation 1 that $||h_{\ell}(\cdot)||_{\infty}\leq LN^{2}V^{2}$. Then we can rewrite the above inequality as

[TABLE]

As a result, we have

[TABLE]

To simplify notations, we define a row vector \mathbf{v}_{\ell}\triangleq\Big{(}v_{\ell}\big{(}\mathbf{Q},\pi_{\ell}(\mathbf{Q})\big{)}\Big{)}_{\mathbf{Q}\in\mathcal{Q}_{V}}, a column vector \mathbf{h}_{\ell}\triangleq\big{(}h_{\ell}(\mathbf{Q})\big{)}_{\mathbf{Q}\in\mathcal{Q}_{V}} and a matrix \mathbf{G}_{\ell}=\Big{(}P_{\ell}\big{(}\mathbf{Q}^{\prime}|\mathbf{Q},\pi_{\ell}(\mathbf{Q})\big{)}\Big{)}_{\mathbf{Q},\mathbf{Q}^{\prime}\in\mathcal{Q}_{V}}. Then (20) can be rewritten as

[TABLE]

The second term in the RHS of (21) can be further decomposed as follows:

[TABLE]

where $\mathbf{G}^{*}_{\ell}$ is the transition matrix if we apply policy $\pi_{\ell}$ in the truncated true MDP model $M_{V}^{*}$. Summing over all episodes $\ell\in\mathcal{L}_{2}$, we have

[TABLE]

Simarly, by Lemma 19 in [2], the first term in the RHS of (23) can be bounded by $O(\sqrt{T})$, which is a lower-order term. Next we find upper bounds for the last two terms in the RHS of (23), given in the following two lemmas.

Lemma 8**.**

$\sum_{\ell\in\mathcal{L}_{2}}\mathbf{v}_{\ell}(\mathbf{I}-\mathbf{G}^{*}_{\ell})\mathbf{h}_{\ell}=O(V^{2}\sqrt{T\log T}+V^{N+2}\log T)$* with probability at least $1-\frac{1}{T}$.*

Proof.

See Appendix H. ∎

Lemma 9**.**

\sum_{\ell\in\mathcal{L}_{2}}\mathbf{v}_{\ell}(\mathbf{G}^{*}_{\ell}-\mathbf{G}_{\ell})\mathbf{h}_{\ell}=O\Big{(}V^{2+N/2}\sqrt{T\log(TV)}\Big{)}.

Proof.

See Appendix I. ∎

As a result, combining Lemma 8 and Lemma 9, we can conclude that with probability at least $1-\frac{1}{T}$

[TABLE]

which completes the proof.

Appendix H Proof to Lemma 8

We find an upper bound for $\sum_{\ell\in\mathcal{L}_{2}}\mathbf{v}_{\ell}(\mathbf{I}-\mathbf{G}^{*}_{\ell})\mathbf{h}_{\ell}$. We first define a suitable martingale and make use of the Azuma-Hoeffding inequality.

Lemma 10** (Azuma-Hoeffding inequality).**

Let $X_{1},X_{2},\cdots$ be a martingale difference sequence with $|X_{i}|\leq c$ for all $i$. Then for any $\epsilon>0$ and $n\in\mathbb{N}$,

[TABLE]

Define a $|\mathcal{Q}_{V}|$-dimensional vector $\mathbf{e}_{Q}$ where the coordinate corresponding to state $\mathbf{Q}$ being 1 and all other coordinates being 0. Let \mathcal{F}_{t}=\big{(}\mathbf{Q}(0),\alpha_{0},\mathbf{Q}(1),\cdots,\mathbf{Q}(t),\alpha_{t}\big{)} be the sequence of states and actions up to slot $t$, and let $\ell(t)$ be the episode containing slot $t$. Consider the sequence

[TABLE]

where $\alpha_{t}=\pi_{\ell}(\mathbf{Q}(t))$ is the action taken by the TUCRL algorithm in slot $t$. Note that conditioned on the historical sequence $\mathcal{F}_{t}$, the randomness of $X_{t}$ is in the next state $\mathbf{Q}(t+1)$. Also, $\{X_{t}\}_{t\geq 0}$ is a sequence of martingale differences since $\mathbb{E}_{Q(t+1)}[X_{t}|\mathcal{F}_{t}]=0$ for all $t$. In addition, we have $|X_{t}|\leq||\mathbf{e}_{Q(t+1)}\mathbf{h}_{\ell}||_{1}\leq||\mathbf{h}_{\ell}||_{\infty}\leq LN^{2}V^{2}$.

Then we have

[TABLE]

where the last inequality is due to $||\mathbf{h}_{\ell}||_{\infty}\leq LN^{2}V^{2}$. Let $\mathcal{T}_{2}$ be the set of slots contained in episodes $\ell\in\mathcal{L}_{2}$. By the Azuma-Hoeffding inequality, we have

[TABLE]

Summing over $\ell\in\mathcal{L}_{2}$, we have with probability at least $1-\frac{1}{T}$

[TABLE]

where we use the fact that $|\mathcal{T}_{2}|\leq T$ and $|\mathcal{L}_{2}|\leq 1+|\mathcal{Q}_{V}||\mathcal{A}|(\log_{2}T+1)$ (see Lemma 5).

Appendix I Proof to Lemma 9

We find an upper bound for $\sum_{\ell\in\mathcal{L}_{2}}\mathbf{v}_{\ell}(\mathbf{G}^{*}_{\ell}-\mathbf{G}_{\ell})\mathbf{h}_{\ell}$. Since both the truncated true MDP $M_{V}^{*}$ and the selected optimistic MDP $M_{\ell}$ are in the confidence set $\mathcal{M}_{\ell}$, the term $\mathbf{G}^{*}_{\ell}-\mathbf{G}_{\ell}$ can be bounded using the confidence set construction (6). We have

[TABLE]

where $P_{V}^{*}$ is the transition matrix for the truncated true MDP $M_{V}^{*}$ and we use the fact that $||\mathbf{h}_{\ell}||_{\infty}\leq LN^{2}V^{2}$.

Let $n(\mathbf{Q},\alpha)=\sum_{\ell}v_{\ell}(\mathbf{Q},\alpha)$ such that $\sum_{(\mathbf{Q},\alpha)}n(\mathbf{Q},\alpha)=T$. Moreover, recall that $n_{\ell}(\mathbf{Q},\alpha)=\sum_{i<\ell}v_{i}(\mathbf{Q},\alpha)$, and that $v_{\ell}(\mathbf{Q},\alpha)\leq n_{\ell}(\mathbf{Q},\alpha)$ by the stopping condition of episode $\ell$. As a result, we have

[TABLE]

Here we used for the second inequality that

[TABLE]

where $X_{\ell}=\max\{1,\sum_{i=1}^{\ell}x_{i}\}$ and $0\leq x_{\ell}\leq X_{\ell-1}$ (see Lemma 19 in [2]), and we used Jensen’s inequality for the last inequality. Finally, by the definition of $\mathcal{L}_{1}$ and $\mathcal{L}_{2}$ we have

[TABLE]

which completes the proof.

Appendix J Proof to Theorem 4

Let $\mathcal{X}$ be the set of slots when $\sum_{i,k}Q_{ik}(t)\geq V-ND$. Note that the amount of exogenous arrivals to each queue is at most $D$ in each time slot, and thus packet dropping occurs in a slot $t$ only if $t\in\mathcal{X}$. As a result, in order to bound the amount of dropped packets, we find an upper bound for $|\mathcal{X}|$.

Note that

[TABLE]

By Theorem 3, with probability at least 1-O\Big{(}\frac{1}{V}+\frac{1}{T}\Big{)}, the TUCRL algorithm achieves

[TABLE]

which implies that

[TABLE]

Since at most $ND$ packets are dropped for each slot $t\in\mathcal{X}$, the amount of dropped packets within $T$ slots is at most

[TABLE]

Note that the amount of total exogenous arrivals $\sum_{i,k}a_{ik}(t)$ in each slot $t$ is i.i.d. with expectation $\mathbb{E}[\sum_{i,k}a_{ik}(t)]=\sum_{i,k}\lambda_{ik}$, and $\sum_{i,k}a_{ik}(t)\in[0,ND]$. By Hoefdding’s inequality, we have

[TABLE]

When $T>\frac{2N^{2}D^{2}\log T}{(\sum_{i,k}\lambda_{ik})^{2}}$, the above probability is less than $\frac{1}{T}$. As a result, with probability at least 1-O\Big{(}\frac{1}{T}+\frac{1}{V}\Big{)}, the fraction of dropped packets is at most

[TABLE]

Appendix K Proof to Lemma 4

By Assumption 1, we have that for any $t\geq 0$

[TABLE]

for some constant $C>0$. Then it follows that for any $T\geq 0$

[TABLE]

By Bounded Convergence Theorem, we have

[TABLE]

As a result, we have

[TABLE]

By Markov inequality, we have

[TABLE]

As a result, with probability at least 1-O\Big{(}\frac{1}{V}\Big{)}, the optimal policy never visits any queue length vector outside $\mathcal{Q}_{V}$ and the optimal average queue length is not affected in the truncated MDP, i.e., $J^{*}_{V}=J^{*}<\infty$, which implies that \mathbb{P}[J_{V}^{*}=J^{*}]\geq 1-O\Big{(}\frac{1}{V}\Big{)}.