Infinite horizon asymptotic average optimality for large-scale parallel server networks

TL;DR

This paper investigates optimal control strategies for large-scale parallel server networks in the Halfin-Whitt regime, establishing asymptotic optimality and proposing stabilizing policies with exponential ergodicity.

Contribution

It introduces a new framework for asymptotic optimality in multi-class, multi-pool server networks and develops state-dependent policies ensuring stability and optimality.

Findings

Optimal values converge to ergodic control solutions.

State-dependent policies stabilize the diffusion process.

Exponential ergodicity under certain abandonment conditions.

Abstract

We study infinite-horizon asymptotic average optimality for parallel server network with multiple classes of jobs and multiple server pools in the Halfin-Whitt regime. Three control formulations are considered: 1) minimizing the queueing and idleness cost, 2) minimizing the queueing cost under a constraints on idleness at each server pool, and 3) fairly allocating the idle servers among different server pools. For the third problem, we consider a class of bounded-queue, bounded-state (BQBS) stable networks, in which any moment of the state is bounded by that of the queue only (for both the limiting diffusion and diffusion-scaled state processes). We show that the optimal values for the diffusion-scaled state processes converge to the corresponding values of the ergodic control problems for the limiting diffusion. We present a family of state-dependent Markov balanced saturation policies (BSPs) that stabilize the controlled diffusion-scaled state processes. It is shown that under these policies, the diffusion-scaled state process is exponentially ergodic, provided that at least one class of jobs has a positive abandonment rate. We also establish useful moment bounds, and study the ergodic properties of the diffusion-scaled state processes, which play a crucial role in proving the asymptotic optimality.

Equations

$$\mathds{R}^{{\mathcal{G}}}_{+}\;\coloneqq\;\bigl{\{}\xi=[\xi_{ij}]\in\mathds{R}^{I\times J}_{+}\colon\xi_{ij}=0~{}~{}\text{for~{}}i\nsim j\bigr{\}}\,,$$

$$\frac{{\lambda }_i^n-n{\lambda }_i}{\sqrt{n}}\to {\hat{\lambda }}_i,\sqrt{n}({\mu }_{ij}^n-{\mu }_{ij})\to {\hat{\mu }}_{ij},\frac{N_j^n-n{\nu }_j}{\sqrt{n}}\to {\hat{\nu }}_j,{\gamma }_i^n\to {\gamma }_i.$$

$$\text{Minimize}\underset{j\in \mathcal{J}}{\max }\sum _{i\in \mathcal{I}}{\xi }_{ij},\text{subject to}\sum _{j\in \mathcal{J}}{\mu }_{ij}{\nu }_j{\xi }_{ij}={\lambda }_i,i\in \mathcal{I},\text{and }[{\xi }_{ij}]\in \text{mathds}{R}_{+}^{\mathcal{G}},$$

$$\sum _{i\in \mathcal{I}}{\xi }_{ij}^{\ast }=1,\forall j\in \mathcal{J},\text{and}{\xi }_{ij}^{\ast }>0\text{for all }i\sim j.$$

$$x_i^{\ast }=\sum _{j\in \mathcal{J}}{\xi }_{ij}^{\ast }{\nu }_j,z_{ij}^{\ast }={\xi }_{ij}^{\ast }{\nu }_j.$$

$$\begin{array}{rl}{X}_i^n(t)& =Q_i^n(t)+\sum _{j\in \mathcal{J}(i)}{Z}_{ij}^n(t)\forall i\in \mathcal{I},\\ N_j^n& =Y_j^n(t)+\sum _{i\in \mathcal{I}(j)}{Z}_{ij}^n(t)\forall j\in \mathcal{J}.\end{array}$$

$$Q_i^n(t)\wedge Y_j^n(t)=0\forall i\sim j,\forall t\ge 0.$$

$$\begin{array}{rl}{q}_i(x,z)& :=x_i-\sum _{j\in \mathcal{J}}{z}_{ij},i\in \mathcal{I},\\ y_j^n(z)& :=N_j^n-\sum _{i\in \mathcal{J}}{z}_{ij},j\in \mathcal{J},\end{array}$$

$${\mathcal{Z}^{n}}(x)\;\coloneqq\;\bigl{\{}z\in\mathds{Z}^{{\mathcal{G}}}_{+}\;\colon q_{i}(x,z)\wedge y_{j}^{n}(z)=0\,,~{}q_{i}(x,z)\geq 0\,,~{}y_{j}^{n}(z)\geq 0\;\;\;\forall\,(i,j)\in{\mathcal{E}}\bigr{\}}\,.$$

$${\mathcal{F}}_t^n$$

$${\mathcal{G}}_t^n$$

$${\tilde{A}}_i^n(t)$$

$${\tilde{S}}_{ij}^n(t)$$

$${\tilde{R}}_i^n(t)$$

$$X_i^n(t)=X_i^n(0)+A_i^n({\lambda }_i^{n}t)-\sum _{j\in \mathcal{J}(i)}{S}_{ij}^n\left({\mu }_{ij}^n{\int }_0^t{Z}_{ij}^n(s)\mathrm{d}s\right)-R_i^n\left({\gamma }_i^n{\int }_0^t{Q}_i^n(s)\mathrm{d}s\right),$$

$$\displaystyle{\mathscr{L}}_{n}^{z}f(x)\;\coloneqq\;\sum_{i\in{\mathcal{I}}}\lambda^{n}_{i}\bigl{(}f(x+e_{i})$$

$$\displaystyle+\sum_{i\in{\mathcal{I}}}\gamma_{i}^{n}q_{i}(x,z)\bigl{(}f(x-e_{i})-f(x)\bigr{)}\,,\qquad f\in\mathcal{C}(\mathds{R}^{I})\,,\quad x\in\mathds{Z}^{I}_{+}\,.$$

$$e\cdot q(x,z)\wedge e\cdot y^n(z)=0.$$

$${\mathscr{X}}^{n}\;\coloneqq\;\bigl{\{}x\in\mathds{Z}^{I}_{+}\,\colon\,~{}e\cdot q(x,z)\wedge e\cdot y^{n}(x,z)\;=\;0~{}\text{for some}~{}z\in{\mathcal{Z}^{n}}(x)\bigr{\}}\,,$$

$$e\cdot (x-N^n)=e\cdot q(x,z)-e\cdot y^n(z),$$

$$e\cdot q(x,z)\,=\,\bigl{[}e\cdot(x-N^{n})\bigr{]}^{+}\,,\quad\text{and}\quad e\cdot y^{n}(z)\,=\,\bigl{[}e\cdot(x-N^{n})\bigr{]}^{-}\,.$$

$$\varTheta^{n}(x)\;\coloneqq\;\bigl{\{}(q,y)\in\mathds{Z}_{+}^{I}\times\mathds{Z}_{+}^{J}\;\colon e\cdot q=[e\cdot(x-N^{n})]^{+}\,,~{}e\cdot y=[e\cdot(x-N^{n})]^{-}\bigr{\}}\,,\qquad x\in\mathds{Z}_{+}^{I}\,.$$

$$D_{\Psi}\;\coloneqq\;\bigl{\{}(\alpha,\beta)\in\mathds{R}^{I}\times\mathds{R}^{J}\;\colon\,e\cdot\alpha=e\cdot\beta\bigr{\}}\,.$$

$$\sum _j{\Psi }_{ij}(\alpha ,\beta )={\alpha }_i\forall i\in \mathcal{I},\text{and}\sum _i{\Psi }_{ij}(\alpha ,\beta )={\beta }_j\forall j\in \mathcal{J},$$

$$\breve{{\mathscr{X}}}^{n}\;\coloneqq\;\bigl{\{}x\in\mathds{Z}^{I}_{+}\;\colon\,\lVert x-nx^{*}\rVert\leq M_{0}\,n\bigr{\}}\,,$$

$$\Psi (x-q,N^n-y)\in \text{mathds}{Z}_{+}^{I\times J}\forall q,y\in {\Theta }^n(x),\forall x\in {\breve{\mathcal{X}}}^n.$$

$${\breve{\mathcal{Z}}^{n}}(x)\;\coloneqq\;\begin{cases}\bigl{\{}z\in{\mathcal{Z}^{n}}(x)\,\colon\,~{}e\cdot q(x,z)\wedge e\cdot y^{n}(z)\;=\;0\bigr{\}}&\text{if\ \ }x\in n(\breve{B}+x^{*})\,,\\[3.0pt] {\mathcal{Z}^{n}}(x)&\text{otherwise,}\end{cases}$$

$$\begin{split}\breve{\mathfrak{Z}}^{n}&\;\coloneqq\;\bigl{\{}Z^{n}\in{\mathfrak{Z}}^{n}\,\colon\,Z^{n}(t)\in{\breve{\mathcal{Z}}^{n}}\bigl{(}X^{n}(t)\bigr{)}\;\;\;\forall\,t\geq 0\bigr{\}}\,,\\[5.0pt] \bm{{\mathfrak{Z}}}&\;\coloneqq\;\{Z^{n}\in\breve{\mathfrak{Z}}^{n}\,,~{}n\in\mathds{N}\}\,.\end{split}$$

$$\begin{array}{rl}{\hat{X}}_i^n(t)& :=\frac{1}{\sqrt{n}}(X_i^n(t)-n{x}_i^{\ast }),\\ {\hat{Q}}_i^n(t)& :=\frac{1}{\sqrt{n}}{Q}_i^n(t),\end{array}\begin{array}{rl}{\hat{Z}}_{ij}^n(t)& :=\frac{1}{\sqrt{n}}(Z_{ij}^n(t)-n{z}_{ij}^{\ast }),\\ {\hat{Y}}_j^n(t)& :=\frac{1}{\sqrt{n}}{Y}_j^n(t).\end{array}$$

$$\begin{array}{rl}{\hat{M}}_{A,i}^n(t)& :=\frac{1}{\sqrt{n}}(A_i^n({\lambda }_i^{n}t)-{\lambda }_i^{n}t),\\ {\hat{M}}_{S,ij}^n(t)& :=\frac{1}{\sqrt{n}}\left(S_{ij}^n\left({\mu }_{ij}^n{\int }_0^t{Z}_{ij}^n(s)\mathrm{d}s\right)-{\mu }_{ij}^n{\int }_0^t{Z}_{ij}^n(s)\mathrm{d}s\right),\\ {\hat{M}}_{R,i}^n(t)& :=\frac{1}{\sqrt{n}}\left(R_i^n\left({\gamma }_i^n{\int }_0^t{Q}_i^n(s)\mathrm{d}s\right)-{\gamma }_i^n{\int }_0^t{Q}_i^n(s)\mathrm{d}s\right).\end{array}$$

$$⟨{\hat{M}}_{A,i}^n⟩(t):=\frac{{\lambda }_i^n}{n}t,⟨{\hat{M}}_{S,ij}^n⟩(t):=\frac{{\mu }_{ij}^n}{n}{\int }_0^t{Z}_{ij}^n(s)\mathrm{d}s,⟨{\hat{M}}_{R,i}^n⟩(t):=\frac{{\gamma }_i^n}{n}{\int }_0^t{Q}_i^n(s)\mathrm{d}s.$$

$$\begin{gathered} {\hat{X}}_i^n(t)={\hat{X}}_i^n(0)+{\ell }_i^{n}t-\sum _{j\in \mathcal{J}(i)}{\mu }_{ij}^n{\int }_0^t{\hat{Z}}_{ij}^n(s)\mathrm{d}s-{\gamma }_i^n{\int }_0^t{\hat{Q}}_i^n(s)\mathrm{d}s \\ +{\hat{M}}^n(t), \end{gathered}$$

Full text

1. Introduction

Large-scale parallel server networks are used to model various service, manufacturing and telecommunications systems; see, e.g., [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13]. We consider multiclass multi-pool networks operating in the Halfin–Whitt (H–W) regime, where the demand of each class and the numbers of servers in each pool get large simultaneously in an appropriate manner so that the system becomes critically loaded while the service and abandonment rates are fixed. We study optimal control problems of such networks under the infinite-horizon expected average (ergodic) cost criteria, since steady-state performance measures are among the most important metrics to understand the system dynamics. Specifically, we consider the following unconstrained and constrained ergodic control problems (see Sections 3.1 and 4.2): (P1) minimizing the queueing and idleness cost, (P2) minimizing the queueing cost while imposing a dynamic constraint on the idleness of each server pool (e.g., requiring that the long-run average idleness does not exceed a given threshold), and (P3) minimizing the queueing cost while requiring fairness on idleness (e.g., the average idleness of each server pool is a fixed proportion of the total average idleness of all server pools). The scheduling policy determines the allocation of service capacity to each class at each time. We consider only work conserving scheduling policies that are non-anticipative and preemptive.

In [14] and [15], we have studied the corresponding ergodic control problems (P1*′)–(P2′) for the limiting diffusions arising from such networks (see Section 3.2). Problem (P1′) for multiclass networks (“V” networks) was studied in [14], where a comprehensive study of the ergodic control problem for a broader class of diffusions as well as asymptotic optimality results for the network model were presented. In [15] we have shown that problem (P1′) and (P2′*) are well-posed for multiclass multi-pool networks, and presented a full characterization of optimality for the limiting diffusion. We also provided important insights on the stabilizability111We say that a control (policy) is stabilizing, if it results in a finite value for the optimization criterion. of the controlled diffusion by employing a leaf elimination algorithm, which is used to derive an explicit expression for the drift. We addressed problem (P3) for the ‘N’ network model, where we studied its well-posedness, characterized the optimal solutions, and established asymptotic optimality in [16]. For this particular network topology, the fairness constraint requires that the long-run average idleness of the two server pools satisfies a fixed ratio condition.

In this paper we establish the asymptotic optimality for the ergodic control problems (P1)–(P3). In other words, we show that the optimal values for the diffusion-scaled state processes converge to the corresponding values for the limiting diffusion. The main challenge lies in understanding the recurrence properties of the diffusion-scaled state processes for multiclass multi-pool networks in the H–W regime. Despite the recent studies on stability of multiclass multi-pool networks under certain scheduling policies [17, 18, 19], the existing results are not sufficient for our purpose. The difficulty is particularly related to the so-called “joint work conservation” (JWC) condition, which requires that no servers are idling unless all the queues are empty, and which plays a key role in the derivation of the limiting diffusion, and the study of discounted control problems in [20, 21]; see Section 2.2 for a detailed discussion. For the limiting diffusion, the JWC condition holds over the entire state space; however, for the diffusion-scaled state process in the $n^{\rm th}$ system ($n$ is the scaling parameter), it holds only in a bounded subset of the state space. As a consequence, a stabilizing control222To avoid confusion, ‘control’ always refers to a control strategy for the limiting diffusion, while ‘policy’ refers to a scheduling strategy for the pre-limit model. for the limiting diffusion cannot be directly translated to a scheduling policy for the $n^{\rm th}$ system which stabilizes the diffusion-scaled state process in the H–W regime.

Our first main contribution addresses the above mentioned critical issue of stabilizability of the diffusion-scaled state processes. We have identified a family of stabilizing policies for multiclass multi-pool networks, which we refer to as the “Balanced Saturation Policies” (BSPs) (see Definition 5.1). Such a policy strives to keep the state process for each class ‘close’ to the corresponding steady state quantity, which is state-dependent and dynamic. The specific stabilizing policy for the ‘N’ network presented in [16] belongs to this family of BSPs. We show that if the abandonment parameter is positive for at least one class, then the diffusion-scaled state processes are exponentially stable under any BSP (see Proposition 5.1).

In addition to the ergodicity properties, moment bounds are also essential to prove asymptotic optimality. An important implication of the exponential ergodicity property proved in [15, Theorem 4.2] is that the controlled diffusion satisfies a very useful moment bound, see Eq. 2.29, namely, that any moment (higher than first order) of the state is controlled by the corresponding moments of the queue and idleness. This moment bound is also shown for the $n^{\rm th}$ system (Proposition 6.1). In studying the moment bounds, we have identified an important class of multiclass multi-pool networks, which we refer to as bounded-queue, bounded-state (BQBS) stable networks (Section 4). The limiting diffusion of this class of networks has the following important property. Any moment (higher than first order) of the state is controlled by the corresponding moment of the queue alone (see Proposition 6.2). The class of BQBS stable networks contains many interesting examples, including networks with a single dominant class (see Figure 1), and networks with certain parameter constraints, e.g., service rates that are only pool-dependent. It is worth noting that the ergodic control problem with fairness constraints for the general multiclass multi-pool networks may not be well-posed. However, the problems (P3) and (P3*′) are well-posed for the BQBS stable networks. In addition, the problems (P1) and (P1′*) which only penalize the queueing cost are also well-posed for the BQBS stable networks.

The proof of asymptotic optimality involves the convergence of the value functions, specifically, establishing the lower and upper bounds (see Theorems 3.1, 3.2 and 4.2). In establishing the lower bound, the arguments are analogous to those for the ‘N’ network in [16] and the ‘V’ network in [14]. This involves proving the tightness of the mean empirical measures of the diffusion-scaled state process, controlled under some eventually JWC scheduling policy (see Definition 2.2), and also showing that any limit of these empirical measures is an ergodic occupation measure for the limiting diffusion model (see Lemma 6.1).

The proof of the upper bound is the most challenging. We utilize the following important property which arises from a spatial truncation technique for all three problems (P1*′)–(P3′*): there exists a continuous precise stationary Markov control $\bar{v}_{\epsilon}$ which is $\epsilon$-optimal for the limiting diffusion control problem, and under which the diffusion is exponentially ergodic (see Lemma 7.1, Corollary 7.1, and [14, Theorem 4.2]). For the $n^{\rm th}$ system, we construct a concatenated admissible policy in the following manner. In the JWC conservation region, we apply a scheduling policy constructed canonically from the Markov control $\bar{v}_{\epsilon}$ (see Definition 6.1), while outside the JWC region, we apply a fixed BSP. We also show that that under this concatenated policy the diffusion-scaled state process is exponentially ergodic, and its mean empirical measures converge to the ergodic occupation measure of the limiting diffusion associated with the control $\bar{v}_{\epsilon}$.

1.1. Literature review

There is an extensive literature on scheduling control of multiclass multi-pool networks in the H–W regime. For the infinite-horizon discounted criterion, Atar [20, 21] first studied the unconstrained scheduling control problem under a set of conditions on the network structure, the system parameters, and the running cost function (Assumptions 2 and 3 in [21]). Atar et al. [22] further investigated simplified models with service rates that either only class-dependent, or pool-dependent. Gurvich and Whitt [23, 24, 25] studied queue-and-idleness-ratio controls for multiclass multi-pool networks, by proving a state-space-collapse (SSC) property under suitable conditions on the network structure and system parameters (Theorems 3.1 and 5.1 in [23]). For finite-horizon cost criteria, Dai and Tezcan [26, 27] studied scheduling controls of multiclass multi-pool networks, also by proving an SSC property under certain assumptions.

There has also been a lot of activity on ergodic control of multiclass multi-pool networks in the H–W regime, in addition to [14, 15, 16] mentioned earlier. For the inverted ‘V’ model, Armony [28] has shown that the fastest-server-first policy is asymptotically optimal for minimizing the steady-state expected queue length and waiting time, and Armony and Ward [29] have shown that a threshold policy is asymptotically optimal for minimizing the expected queue length and waiting time subject to a “fairness” constraint on the workload division. For multiclass multi-pool networks, Ward and Armony [30] have studied blind fair routing policies, and used simulations to validate their performance, and compared them with non-blind policies derived from the limiting diffusion control problem. Biswas [31] recently studied a specific multiclass multi-pool network with “help” where each server pool has a dedicated stream of a customer class, and can help with other customer classes only when it has idle servers. For this network model, the control policies may not be work-conserving, and the associated controlled diffusion has a uniform stability property, which is not satisfied for general multiclass multi-pool networks.

This work contributes to the understanding of the stability of multiclass multi-pool networks in the H-W regime. Gamarnik and Stolyar [32] studied the tightness of the stationary distributions of the diffusion-scaled state processes under any work conserving scheduling policy for the ‘V’ network, while ergodicity properties for the limiting diffusion under constant Markov controls are established in [33, 34]. We refer the reader to [18, 17, 19] for the stability analysis of a load balancing scheduling policy, “longest-queue freest-server” (LQFS-LB), and a leaf activity priority policy, for multiclass multi-pool networks. For the ‘N’ network with no abandonment, Stolyar [35] studied the stability of a static priority scheduling policy.

1.2. Organization of the paper

In the subsection which follows we summarize the notation used in the paper. In Section 2.1, we describe the model and the scheduling control problems, and in Section 2.2, we discuss the JWC condition. In Section 2.3, we state some basic properties of the diffusion-scaled processes and the control parameterization, which leads to the diffusion limit. In Section 2.4, we review some relevant properties of the limiting diffusion from [15]. We state the control objectives of the problems (P1) and (P2) in Section 3.1, and the corresponding diffusion control problems (P1*′) and (P2′) in Section 3.2, and summarize the asymptotic optimality results in Section 3.3. In Section 4 we describe the BQBS stable networks and study the fairness problems (P3) and (P3′*). In Section 5, we introduce the family of stabilizing BSPs, and show that under these we have exponential stability. In Section 6, we focus on the ergodic properties of the $n^{\rm th}$ system, including the moment bounds, convergence of mean empirical measures and a stability preserving property in the JWC region. In Section 7, we complete the proofs of the lower and upper bounds of the three problems. We conclude in Section 8.

1.3. Notation

The symbol $\mathds{R}$ denotes the field of real numbers, and $\mathds{R}_{+}$ and $\mathds{N}$ denote the sets of nonnegative real numbers and natural numbers, respectively. The minimum (maximum) of two real numbers $a$ and $b$, is denoted by $a\wedge b$ ($a\vee b$). Define $a^{+}\coloneqq a\vee 0$ and $a^{-}\coloneqq-(a\wedge 0)$. The integer part of a real number $a$ is denoted by $\lfloor a\rfloor$. We also let $e\coloneqq(1,\dotsc,1)^{\mathsf{T}}$.

For a set $A\subset\mathds{R}^{d}$, we use $\bar{A}$, $A^{c}$, and $\mathds{1}_{A}$ to denote the closure, the complement, and the indicator function of $A$, respectively. A ball of radius $r>0$ in $\mathds{R}^{d}$ around a point $x$ is denoted by $B_{r}(x)$, or simply as $B_{r}$ if $x=0$. The Euclidean norm on $\mathds{R}^{d}$ is denoted by $\lvert\,\cdot\,\rvert$, $x\cdot y$ denotes the inner product of $x,y\in\mathds{R}^{d}$, and $\lVert x\rVert\coloneqq\sum_{i=1}^{d}\lvert x_{i}\rvert$.

We let $\mathcal{C}^{\infty}_{c}(\mathds{R}^{d})$ denote the set of smooth real-valued functions on $\mathds{R}^{d}$ with compact support. For a Polish space ${\mathcal{X}}$, we denote by ${\mathcal{P}}({\mathcal{X}})$ the space of probability measures on the Borel subsets of ${\mathcal{X}}$ under the Prokhorov topology. For $\nu\in{\mathcal{P}}({\mathcal{X}})$ and a Borel measurable map $f\colon{\mathcal{X}}\to\mathds{R}$, we often use the abbreviated notation $\nu(f)\coloneqq\int_{{\mathcal{X}}}f\,\mathrm{d}{\nu}\,.$ The quadratic variation of a square integrable martingale is denoted by $\langle\,\cdot\,,\cdot\,\rangle$. For any path $X(\cdot)$ of a càdlàg process, we use the notation $\Delta X(t)$ to denote the jump at time $t$.

2. The Model

All random variables introduced below are defined on a complete probability space $(\Omega,\mathfrak{F},\operatorname{\mathbb{P}})$ and $\operatorname{\mathbb{E}}$ denotes the associated expectation operator.

2.1. The multiclass multi-pool network model

We consider a sequence of network systems with the associated variables, parameters and processes indexed by $n$. Each of these, is a multiclass multi-pool Markovian network with $I$ classes of customers and $J$ server pools, labeled as $1,\dots,I$ and $1,\dots,J$, respectively. Let ${\mathcal{I}}=\{1,\dots,I\}$ and ${\mathcal{J}}=\{1,\dots,J\}$. Customers of each class form their own queue and are served in the first-come-first-served (FCFS) service discipline. The buffers of all classes are assumed to have infinite capacity. Customers can abandon/renege while waiting in queue. Each class of customers can be served by a subset of server pools, and each server pool can serve a subset of customer classes. We let ${\mathcal{J}}(i)\subset{\mathcal{J}}$, denote the subset of server pools that can serve class $i$ customers, and ${\mathcal{I}}(j)\subset{\mathcal{I}}$ the subset of customer classes that can be served by server pool $j$. We form a bipartite graph ${\mathcal{G}}=({\mathcal{I}}\cup{\mathcal{J}},{\mathcal{E}})$ with a set of edges defined by ${\mathcal{E}}=\{(i,j)\in{\mathcal{I}}\times{\mathcal{J}}\colon j\in{\mathcal{J}}(i)\}$, and use the notation $i\sim j$, if $(i,j)\in{\mathcal{E}}$, and $i\nsim j$, otherwise. We assume that the graph ${\mathcal{G}}$ is a tree.

For each $j\in{\mathcal{J}}$, let $N_{j}^{n}$ be the number of servers (statistically identical) in server pool $j$. Set $N^{n}=(N^{n}_{j})_{j\in{\mathcal{J}}}$. Customers of class $i\in{\mathcal{I}}$ arrive according to a Poisson process with rate $\lambda^{n}_{i}>0$, and have class-dependent exponential abandonment rates $\gamma_{i}^{n}\geq 0$. These customers are served at an exponential rate $\mu_{ij}^{n}>0$ at server pool $j$, if $i\sim j$, and we set $\mu_{ij}^{n}=0$, if $i\nsim j$. Thus, the set of edges ${\mathcal{E}}$ can thus be written as {\mathcal{E}}=\bigl{\{}(i,j)\in{\mathcal{I}}\times{\mathcal{J}}\;\colon\,\mu_{ij}^{n}>0\bigr{\}}. We assume that the customer arrival, service, and abandonment processes of all classes are mutually independent. We define

[TABLE]

and analogously define $\mathds{Z}^{{\mathcal{G}}}_{+}$.

2.1.1. The Halfin–Whitt regime

We study these multiclass multi-pool networks in the Halfin–Whitt regime (or the Quality-and-Efficiency-Driven (QED) regime), where the arrival rates of each class and the numbers of servers of each server pool grow large as $n\to\infty$ in such a manner that the system becomes critically loaded. Throughout the paper, the set of parameters is assumed to satisfy the following.

Parameter Scaling. There exist positive constants $\lambda_{i}$ and $\nu_{j}$, nonnegative constants $\gamma_{i}$ and $\mu_{ij}$, with $\mu_{ij}>0$ for $i\sim j$ and $\mu_{ij}=0$ for $i\nsim j$, and constants $\hat{\lambda}_{i}$, $\hat{\mu}_{ij}$ and $\hat{\nu}_{j}$, such that the following limits exist as $n\to\infty$.

[TABLE]

Fluid scale equilibrium. We assume that the linear program (LP) given by

[TABLE]

has a unique solution $\xi^{*}=[\xi^{*}_{ij}]\in\mathds{R}^{{\mathcal{G}}}_{+}$ satisfying

[TABLE]

This assumption is referred to as the complete resource pooling condition [36, 21]. It implies that the graph ${\mathcal{G}}$ is a tree [36, 21].

We define $x^{*}=(x_{i}^{*})_{i\in{\mathcal{I}}}\in\mathds{R}^{I}_{+}$, and $z^{*}=[z_{ij}^{*}]\in\mathds{R}^{{\mathcal{G}}}_{+}$ by

[TABLE]

The vector $x^{*}$ can be interpreted as the steady-state total number of customers in each class, and the matrix $z^{*}$ as the steady-state number of customers in each class receiving service, in the fluid scale. Note that the steady-state queue lengths are all zero in the fluid scale. The quantity $\xi_{ij}^{*}$ can be interpreted as the steady-state fraction of service allocation of pool $j$ to class-$i$ jobs in the fluid scale. It is evident that Eqs. 2.2 and 2.3 imply that $e\cdot x^{*}=e\cdot\nu$, where $\nu\coloneqq(\nu_{j})_{j\in{\mathcal{J}}}$.

2.1.2. The state descriptors

For $i\in{\mathcal{I}}$, let $X^{n}_{i}=\{X^{n}_{i}(t)\colon t\geq 0\}$ and $Q^{n}_{i}=\{Q^{n}_{i}(t)\colon t\geq 0\}$ be the number of class $i$ customers in the system and in the queue, respectively, and for $j\in{\mathcal{J}}$, let $Y^{n}_{j}=\{Y^{n}_{j}(t)\colon t\geq 0\}$, be the number of idle servers in pool $j$. We also let $Z_{ij}^{n}=\{Z_{ij}^{n}(t)\colon t\geq 0\}$ denote the number of class $i$ customers being served in server pool $j$. Set $X^{n}=(X_{i}^{n})_{i\in{\mathcal{I}}}$, $Y^{n}=(Y_{j}^{n})_{j\in{\mathcal{J}}}$, $Q^{n}=(Q_{i}^{n})_{i\in{\mathcal{I}}}$, and $Z^{n}=(Z_{ij}^{n})_{i\in{\mathcal{I}},\,j\in{\mathcal{J}}}$. For each $t\geq 0$, we have the fundamental balance equations

[TABLE]

2.1.3. Scheduling control

The control process is $Z^{n}$. We only consider work conserving scheduling policies that are non-anticipative and preemptive. Work conservation requires that the processes $Q^{n}$ and $Y^{n}$ satisfy

[TABLE]

In other words, whenever there are customers waiting in queues, if a server becomes free and can serve one of the customers, the server cannot idle and must decide which customer to serve and start service immediately. Service preemption is allowed, that is, service of a customer can be interrupted at any time to serve some other customer of another class and resumed at a later time.

For $(x,z)\in\mathds{Z}_{+}^{I}\times\mathds{Z}^{{\mathcal{G}}}_{+}$, we define

[TABLE]

and the action set ${\mathcal{Z}^{n}}(x)$ by

[TABLE]

We denote $y_{j}(x,z)\,=\,y_{j}^{n}(x,z)$ whenever no confusion occurs.

Let $A^{n}_{i}$, $S^{n}_{ij}$, and $R^{n}_{i}$, $(i,j)\in{\mathcal{E}}$, be mutually independent rate-$1$ Poisson processes, and also independent of the initial condition $X^{n}_{i}(0)$. Define the $\sigma$-fields

[TABLE]

where $\mathcal{N}$ is the collection of all $\operatorname{\mathbb{P}}$-null sets, and

[TABLE]

The filtration $\bm{\mathscr{F}}^{n}\coloneqq\{\mathscr{F}^{n}_{t}\colon t\geq 0\}$ represents the information available up to time $t$, and the filtration $\bm{\mathscr{G}}^{n}\coloneqq\{\mathscr{G}^{n}_{t}\colon t\geq 0\}$ contains the information about future increments of the processes.

We say that a scheduling policy $Z^{n}$ is admissible if

1. (i)

$Z^{n}(t)\in{\mathcal{Z}^{n}}(X^{n}(t))$ a.s. for all $t\geq 0$; 2. (ii)

$Z^{n}(t)$ is adapted to $\mathscr{F}^{n}_{t}$; 3. (iii)

$\mathscr{F}^{n}_{t}$ is independent of $\mathscr{G}^{n}_{t}$ at each time $t\geq 0$; 4. (iv)

for each $i\in{\mathcal{I}}$ and $i\in{\mathcal{J}}$, and for each $t\geq 0$, the process $\delta\tilde{S}^{n}_{ij}(t,\cdot)$ agrees in law with $S^{n}_{ij}(\mu_{ij}^{n}\,\cdot)$, and the process $\delta\tilde{R}^{n}_{i}(t,\cdot)$ agrees in law with $R^{n}_{i}(\gamma_{i}^{n}\cdot)$.

We denote the set of all admissible scheduling policies $(Z^{n},\bm{\mathscr{F}}^{n},\bm{\mathscr{G}}^{n})$ by ${\mathfrak{Z}}^{n}$. Abusing the notation we sometimes denote this as $Z^{n}\in{\mathfrak{Z}}^{n}$. An admissible policy is called stationary Markov if $Z^{n}(t)=z(X^{n}(t))$ for some function $z\colon\mathds{Z}_{+}^{I}\to\mathds{Z}^{{\mathcal{G}}}_{+}$, in which case we identify the policy with the function $z$.

Under an admissible scheduling policy, the state process $X^{n}$ can be represented as

[TABLE]

for $i\in{\mathcal{I}}$ and $t\geq 0$. Under a stationary Markov policy, $X^{n}$ is Markov with generator

[TABLE]

2.2. Joint work conservation

Definition 2.1**.**

We say that an action $z\in{\mathcal{Z}^{n}}(x)$ is jointly work conserving (JWC), if

[TABLE]

We define

[TABLE]

with $q$ and $y^{n}$ defined in Eq. 2.5.

Since Eq. 2.4 implies that

[TABLE]

it is clear that Eq. 2.8 is satisfied if and only if

[TABLE]

Let

[TABLE]

It is evident that the JWC condition can be met at any point $x\in\mathds{Z}_{+}^{I}$ at which the image of ${\mathcal{Z}^{n}}(x)$ under the map z\mapsto\bigl{(}q(x,z),y^{n}(z)\bigr{)} defined in Eq. 2.5 intersects $\varTheta^{n}(x)$.

Let

[TABLE]

As shown in Proposition A.2 of [20], provided that ${\mathcal{G}}$ is a tree, there exists a unique linear map $\Psi=[\Psi_{ij}]\colon D_{\Psi}\to\mathds{R}^{I\times J}$ solving

[TABLE]

with $\Psi_{ij}(\alpha,\beta)\;=\;0$ for $i\nsim j$.

We quote a result from [21], which is used later. The proof of Lemma 3 in [21] assumes that the limits in Eq. 2.1 exist, in particular, $\sqrt{n}(N_{j}^{n}-n\nu_{j})\to 0$ as $n\to\infty$. Nevertheless, the proof goes through under the weaker assumption that $N_{j}^{n}-n\nu_{j}={\mathfrak{o}}(n)$.

Lemma 2.1** (Lemma 3 in [21]).**

There exists a constant $M_{0}>0$ such that, the collection of sets $\breve{{\mathscr{X}}}^{n}$ defined by

[TABLE]

satisfies $\breve{{\mathscr{X}}}^{n}\subset{\mathscr{X}}^{n}$ for all $n\in\mathds{N}$. Moreover,

[TABLE]

Remark 2.1*.*

Lemma 2.1 implies that if $x\in\breve{{\mathscr{X}}}^{n}$, then for any $q\in\mathds{Z}^{I}_{+}$ and $y\in\mathds{Z}^{J}_{+}$ satisfying $e\cdot q\wedge e\cdot y=0$ and $e\cdot(x-q)=e\cdot(N^{n}-y)\geq 0$, we have $\Psi(x-q,N^{n}-y)\in{\mathcal{Z}^{n}}(x)$.

We need the following definition.

Definition 2.2**.**

We fix some open ball $\breve{B}$ centered at the origin, such that $n(\breve{B}+x^{*})\subset\breve{{\mathscr{X}}}^{n}$ for all $n\in\mathds{N}$. The jointly work conserving action set ${\breve{\mathcal{Z}}^{n}}(x)$ at $x$ is defined as the subset of ${\mathcal{Z}^{n}}(x)$, which satisfies

[TABLE]

with $q$ and $y^{n}$ as in Eq. 2.5. We also define the associated admissible policies by

[TABLE]

We refer to the policies in $\bm{{\mathfrak{Z}}}$ as eventually jointly work conserving (EJWC).

The ball $\breve{B}$ is fixed in Definition 2.2 only for convenience. We could instead adopt a more general definition of $\bm{{\mathfrak{Z}}}$, as explained in Remark 2.1 in [15]. The EJWC condition plays a crucial role in the derivation of the controlled diffusion limit. Therefore, the convergence of mean empirical measures of the controlled diffusion-scaled state process, and thus, also the lower and upper bounds for asymptotic optimality are established for sequences $\{Z^{n},\,n\in\mathds{N}\}\subset\bm{{\mathfrak{Z}}}$.

2.3. The diffusion-scaled processes

Let $x^{*}$ and $z^{*}$ be as in Eq. 2.3. We define the diffusion-scaled processes $\hat{Z}^{n}$, $\hat{X}^{n}$, $\hat{Q}^{n}$, and $\hat{Y}^{n}$, by

[TABLE]

Let

[TABLE]

These are square integrable martingales w.r.t. the filtration $\bm{\mathscr{F}}^{n}$, with quadratic variations

[TABLE]

Let $\widehat{M}^{n}(t)\coloneqq\hat{M}^{n}_{A,i}(t)-\sum_{j\in{\mathcal{J}}(i)}\hat{M}^{n}_{S,ij}(t)-\hat{M}^{n}_{R,i}(t)$. By Eq. 2.6, we can write $\hat{X}^{n}_{i}(t)$ as

[TABLE]

where $\ell^{n}=(\ell^{n}_{1},\dotsc,\ell^{n}_{I})^{\mathsf{T}}$ is defined as

[TABLE]

Under the assumptions on the parameters in Eq. 2.1 and the first constraint in the LP, it holds that

[TABLE]

We let $\ell\coloneqq(\ell_{1},\dotsc,\ell_{I})^{\mathsf{T}}$.

By Eqs. 2.3, 2.4 and 2.11, we obtain the balance equations

[TABLE]

Definition 2.3**.**

For each $x\in\mathds{Z}^{I}_{+}$ and $z\in{\mathcal{Z}^{n}}(x)$, we define

[TABLE]

with $q(x,z)$, $y^{n}(z)$ as in Eq. 2.5. We also let

[TABLE]

and

[TABLE]

Abusing the notation, we also write

[TABLE]

and

[TABLE]

Lemma 2.2**.**

There exists a constant $\tilde{M}_{0}>0$ such that for any $z\in{\breve{\mathcal{Z}}^{n}}(x)$, $x\in\mathds{Z}^{I}_{+}$, and $n\in\mathds{N}$, we have

[TABLE]

Proof.

Note that

[TABLE]

for all $x\in\mathds{Z}^{I}_{+}$ and $z\in{\mathcal{Z}^{n}}(x)$. Therefore, there exist probability vectors $p^{c}\in[0,1]^{I}$ and $p^{s}\in[0,1]^{J}$ such that \hat{q}^{n}=\bigl{(}\hat{\vartheta}^{n}+(e\cdot\hat{x}^{n})^{+}\bigr{)}p^{c} and \hat{y}^{n}=\bigl{(}\hat{\vartheta}^{n}+(e\cdot\hat{x}^{n})^{-}\bigr{)}p^{s}. By the linearity of the map $\Psi$ and Lemma 2.1, it easily follows that

[TABLE]

If $x\notin\breve{\mathscr{X}}^{n}$, then $\lVert\hat{x}^{n}\rVert>M_{0}\sqrt{n}$ by Eq. 2.10. Since for some constant $C>0$, it holds that $\lVert\hat{y}^{n}(z)\rVert\leq C\sqrt{n}$ for all $n\in\mathds{N}$, the same bound also holds for $\hat{\vartheta}^{n}(x,z)$. Thus if $x\notin\breve{\mathscr{X}}^{n}$, we obtain the bound asserted in the lemma by Eqs. 2.17 and 2.18.

On the other hand, if $x\in\breve{\mathscr{X}}^{n}$ and $z\in{\breve{\mathcal{Z}}^{n}}(x)$, then $\hat{\vartheta}^{n}(x,z)=0$, and again the assertion of the lemma follows by Eqs. 2.17 and 2.18. This completes the proof. ∎

Definition 2.4**.**

We define the operator $\mathscr{A}^{n}\colon\mathcal{C}^{2}(\mathds{R}^{I})\to\mathcal{C}(\mathds{R}^{I},\mathds{R}^{I\times J})$ by

[TABLE]

where $\partial_{i}\coloneqq\tfrac{\partial~{}}{\partial{x}_{i}}$ and $\partial_{ij}\coloneqq\tfrac{\partial^{2}~{}}{\partial{x}_{i}\partial{x}_{j}}$, and

[TABLE]

By the Kunita–Watanabe formula for semi-martingales (see, e.g., [37, Theorem 26.7]), we have

[TABLE]

for any admissible diffusion-scaled policy $\hat{Z}^{n}$, where

[TABLE]

2.3.1. Control parameterization

Definition 2.5**.**

Let $\bm{\mathfrak{X}}^{n}\coloneqq\{(\hat{x},\hat{z})\colon\hat{x}\in{\mathscr{S}}^{n}\,,\ \hat{z}\in{\hat{\mathcal{Z}}^{n}}(\hat{x})\}$. For each $(\hat{x},\hat{z})\in\bm{\mathfrak{X}}^{n}$, we define

[TABLE]

and

[TABLE]

Let $u(\hat{x},\hat{z})\coloneqq(u^{c}(\hat{x},\hat{z}),u^{s}(\hat{z}))$. Then $u(\hat{x},\hat{z})$ belongs to the set

[TABLE]

We also define the processes

[TABLE]

and $U^{n}\coloneqq(U^{c,n},U^{s,n})$, with $U^{c,n}\coloneqq(U^{c,n}_{1},\dotsc,U^{c,n}_{I})^{\mathsf{T}}$, and $U^{s,n}\coloneqq(U^{s,n}_{1},\dotsc,U^{s,n}_{J})^{\mathsf{T}}$.

The process $U^{c,n}_{i}(t)$ represents the proportion of the total queue length in the network at queue $i$ at time $t$, while $U^{s,n}_{j}(t)$ represents the proportion of the total idle servers in the network at station $j$ at time $t$. Given $Z^{n}\in{\mathfrak{Z}}^{n}$ the process $U^{n}$ is uniquely determined and lives in the set ${\mathbb{U}}$.

For $u\in{\mathbb{U}}$, let $\widehat{\Psi}[u]\colon\mathds{R}^{I}\to\mathds{R}^{{\mathcal{G}}}$ be defined by

[TABLE]

where $\Psi$ is as in Eq. 2.9.

We define the operator $\breve{\mathscr{A}}^{n}\colon\mathcal{C}^{2}(\mathds{R}^{I})\to\mathcal{C}(\mathds{R}^{I},{\mathbb{U}})$ by

[TABLE]

where

[TABLE]

The following lemma is a result of a simple calculation based on the definitions above. Recall the definitions of $\breve{B}$, ${\breve{\mathcal{Z}}^{n}}$, and $\breve{\mathscr{S}}^{n}$ from Definitions 2.2 and 2.3.

Lemma 2.3**.**

Let $u=u(\hat{x},\hat{z})\colon\bm{\mathfrak{X}}^{n}\to{\mathbb{U}}$ denote the map given in Definition 2.5. Then for $f\in\mathcal{C}^{2}_{c}(\sqrt{n}\breve{B})$, we have

[TABLE]

2.4. The diffusion limit

Consider the $I$-dimensional controlled diffusion given by the Itô equation

[TABLE]

where $W$ is an $I$-dimensional standard Wiener process. The drift $b\colon\mathds{R}^{I}\times{\mathbb{U}}\to\mathds{R}^{I}$ takes the form

[TABLE]

where $\widehat{\Psi}_{ij}[u]$ is as in Eq. 2.22. Also \Sigma\;\coloneqq\;\operatorname*{diag}\bigl{(}\sqrt{2\lambda_{1}},\dotsc,\sqrt{2\lambda_{I}}\bigr{)}.

The control process $U$ takes values in ${\mathbb{U}}$, defined in Eq. 2.21, and $U_{t}(\omega)$ is jointly measurable in $(t,\omega)\in[0,\infty)\times\Omega$. Moreover, it is non-anticipative, i.e., for $s<t$, $W_{t}-W_{s}$ is independent of

[TABLE]

Such a process $U$ is called an admissible control. Let $\mathfrak{U}$ denote the set of all admissible controls. Recall that a control is called Markov if $U_{t}=v(t,X_{t})$ for a measurable map $v\colon\mathds{R}_{+}\times\mathds{R}^{I}\to{\mathbb{U}}$, and it is called stationary Markov if $v$ does not depend on $t$, i.e., $v\colon\mathds{R}^{I}\to{\mathbb{U}}$. Let $\mathfrak{U}_{\mathrm{SM}}$ denote the set of stationary Markov controls. Recall also that a control $v\in\mathfrak{U}_{\mathrm{SM}}$ is called stable if the controlled process is positive recurrent. We denote the set of such controls by $\mathfrak{U}_{\mathrm{SSM}}$. Let

[TABLE]

denote the extended controlled generator of the diffusion in Eq. 2.24.

In [15], a leaf elimination algorithm was developed to obtain an explicit expression for the drift $b(x,u)$. This plays an important role in understanding the recurrence properties of the controlled diffusion. See also Remark 4.2 and Example 4.4 in [15]. We quote this result as follows.

Lemma 2.4** (Lemma 4.3 in [15]).**

The drift $b(x,u)=b(x,(u^{c},u^{s}))$ in the limiting diffusion $X$ in Eq. 2.24 can be expressed as

[TABLE]

where $B_{1}$ is a lower-diagonal $I\times I$ matrix with positive diagonal elements, $B_{2}$ is an $I\times J$ matrix and $\varGamma=\operatorname*{diag}\{\gamma_{1},\dotsc,\gamma_{I}\}$.

The drift in Eq. 2.26 takes the form

[TABLE]

where $j_{i}\in{\mathcal{J}}$, $i\sim j_{i}$, is the unique server-pool node corresponding to $i$ when customer node $i$ is removed by the leaf elimination algorithm (see Section 4.1 in [15]). Two things are important to note: (a) $\tilde{F}_{i}$ is a linear function, and (b) $\mu_{ij_{i}}>0$ (since $i\sim j_{i}$).

Under EJWC policies, convergence in distribution of the diffusion-scaled processes $\hat{X}^{n}$ to the limiting diffusion $X$ in Eq. 2.24 follows by [21, Proposition 3] for certain classes of networks. The fact that Eq. 2.24 can be viewed as a limit of the diffusion-scaled process $\hat{X}^{n}$ is also indicated by the following lemma.

Lemma 2.5**.**

We have

[TABLE]

for $i\in{\mathcal{I}}$, uniformly over compact sets of $\mathds{R}^{I}\times{\mathbb{U}}$. In particular, for any $f\in\mathcal{C}^{2}_{c}(\mathds{R}^{I})$ it holds that

[TABLE]

Nevertheless, for the time being, we consider solutions of Eq. 2.24 as the formal limit of Eq. 2.12. Precise links of the $n^{\text{th}}$ system model and Eq. 2.24 are established in Section 6.

Definition 2.6**.**

Let $\lVert x\rVert_{\beta}\coloneqq(\beta_{1}\,\lvert x_{1}\rvert^{2}+\dotsb+\beta_{I}\,\lvert x_{I}\rvert^{2})^{\nicefrac{{1}}{{2}}}$, with $\beta=(\beta_{1},\dotsc,\beta_{I})$ a positive vector. Throughout the paper, ${\mathcal{V}}_{\kappa,\beta}$, $\kappa\geq 1$, stands for a $\mathcal{C}^{2}(\mathds{R}^{I})$ function which agrees with $\lVert x\rVert_{\beta}^{\kappa}$ on the complement of the unit ball $B$ in $\mathds{R}^{I}$, i.e., ${\mathcal{V}}_{\kappa,\beta}(x)\;=\;\lVert x\rVert_{\beta}^{\kappa}$, for $x\in B^{c}$. Also, $\widetilde{\mathcal{V}}_{\epsilon,\beta}$, $\epsilon>0$, is defined by

[TABLE]

In addition, for $\delta>0$, we define

[TABLE]

As shown in Theorem 4.1 of [15], the drift $b$ in Eq. 2.27 has the following important structural property. For any $\kappa\geq 1$, there exists a function ${\mathcal{V}}_{\kappa,\beta}$ as in Definition 2.6, and positive constants $c_{i}$, $i=0,1,2$, such that

[TABLE]

Since the diffusion matrix is constant, it is evident that a similar estimate holds for $\mathcal{L}^{u}{\mathcal{V}}_{\kappa,\beta}$ uniformly over $u\in{\mathbb{U}}$. By a straightforward application of Itô’s formula, this implies that for any $\kappa\geq 1$ there exists a constant $C$ depending only on $\kappa$ such that (see [15, Lemma 3.1 (c)])

[TABLE]

Moreover, it is shown in [15, Theorem 4.2] that there exists a stationary Markov control $\bar{v}\in\mathfrak{U}_{\mathrm{SM}}$ satisfying

[TABLE]

for any $\kappa\geq 1$, and positive constants $\bar{c}_{0}$ and $\bar{c}_{1}$ depending only on $\kappa$. As a consequence of Eq. 2.30, the diffusion under the control $\bar{v}$ is exponentially ergodic. A slight modification of that proof leads to the following theorem.

Theorem 2.1**.**

Provided that $\gamma_{i}>0$ for some $i\in{\mathcal{I}}$, there exist $\epsilon>0$, a positive vector $\beta\in\mathds{R}^{I}$, and a stationary Markov control $\bar{v}\in\mathfrak{U}_{\mathrm{SM}}$ satisfying

[TABLE]

for some positive constants $\tilde{c}_{0}$ and $\tilde{c}_{1}$.

The properties in Eqs. 2.28, 2.30 and 2.31 are instrumental in showing that the optimal control problems defined in this paper are well posed.

3. Ergodic Control Problems

In this section, we consider two control objectives, which address the queueing (delay) and/or idleness costs in the system: (i) unconstrained problem, minimizing the queueing and idleness cost and (ii) constrained problem, minimizing the queueing cost while imposing a constraint on idleness. We state both problems for the $n^{\rm th}$ system and the limiting diffusion.

3.1. Ergodic control problems for the $n^{\rm th}$ system

The running cost is a function of the diffusion-scaled processes, which are related to the unscaled ones by Eq. 2.11. For simplicity, in all three cost minimization problems, we assume that the initial condition $X^{n}(0)$ is deterministic and $\hat{X}^{n}(0)\to x\in\mathds{R}^{I}$ as $n\to\infty$. Let the running cost $\hat{r}\colon\mathds{R}^{I}_{+}\times\mathds{R}^{J}_{+}\to\mathds{R}_{+}$ be defined by

[TABLE]

where $\xi=(\xi_{1},\dotsc,\xi_{I})^{\mathsf{T}}$ is a positive vector and $\zeta=(\zeta_{1},\dotsc,\zeta_{J})^{\mathsf{T}}$ is a nonnegative vector. In the case $\zeta\equiv 0$, only the queueing cost is minimized. We denote by $\operatorname{\mathbb{E}}^{Z^{n}}$ the expectation operator under an admissible policy $Z^{n}$.

• (P1)

(unconstrained problem) The running cost penalizes the queueing and idleness. Let $\hat{r}(q,y)$ be the running cost function as defined in Eq. 3.1. Here $\zeta>0$. Given an initial state $X^{n}(0)$, and an admissible scheduling policy $Z^{n}\in\breve{\mathfrak{Z}}^{n}$, we define the diffusion-scaled cost criterion by

[TABLE]

The associated cost minimization problem becomes

[TABLE]

• (P2)

(constrained problem) The objective here is to minimize the queueing cost while imposing idleness constraints on the server pools. Let $\hat{r}_{\mathsf{o}}(q)$ be the running cost function corresponding to $\hat{r}$ in Eq. 3.1 with $\zeta\equiv 0$. The diffusion-scaled cost criterion J_{\mathsf{o}}\bigl{(}\hat{X}^{n}(0),Z^{n}\bigr{)} is defined analogously to Eq. 3.2 with running cost $\hat{r}_{\mathsf{o}}(\hat{Q}^{n}(s))$, that is,

[TABLE]

with $\tilde{m}\geq 1$. The associated cost minimization problem becomes

[TABLE]

where $\updelta=(\updelta_{1},\dotsc,\updelta_{J})^{\mathsf{T}}$ is a positive vector.

We refer to $\hat{V}^{n}(\hat{X}^{n}(0))$ and $\hat{V}^{n}_{\mathsf{c}}(\hat{X}^{n}(0))$ as the diffusion-scaled optimal values for the $n^{\rm th}$ system given the initial state $X^{n}(0)$, for (P1) and (P2), respectively.

Remark 3.1*.*

We choose running costs of the form Eq. 3.1 mainly to simplify the exposition. However, all the results of this paper still hold for more general classes of functions. Let $h_{\mathsf{o}}\colon\mathds{R}^{I}\to\mathds{R}_{+}$ be a convex function satisfying $h_{\mathsf{o}}(x)\geq c_{1}\lvert x\rvert^{m}+c_{2}$ for some $m\geq 1$ and constants $c_{1}>0$ and $c_{2}\in\mathds{R}$, and $h\colon\mathds{R}^{I}\to\mathds{R}_{+}$, $h_{i}\colon\mathds{R}\to\mathds{R}_{+}$, $i\in{\mathcal{I}}$, be convex functions that have at most polynomial growth. Then we can choose $\hat{r}(q,y)=h_{\mathsf{o}}(q)+h(y)$ for the unconstrained problem, and $h_{i}(y_{i})$ as the functions in the constraints in Eq. 3.3 (with $\hat{r}_{\mathsf{o}}=h_{\mathsf{o}}$). Analogous running costs can of course be used in the corresponding control problems for the limiting diffusion, which are presented later in Section 3.2.

3.2. Ergodic control problems for the limiting diffusion

We state the two problems which correspond to (P1)–(P2) in Section 3.1 for the controlled diffusion in Eq. 2.24. Let $r\colon\mathds{R}^{I}\times{\mathbb{U}}\to\mathds{R}$ be defined by

[TABLE]

with $\hat{r}$ as in Eq. 3.1, that is,

[TABLE]

for the given $\xi=(\xi_{1},\dotsc,\xi_{I})^{\mathsf{T}}$ and $\zeta=(\zeta_{1},\dotsc,\zeta_{J})^{\mathsf{T}}$ in Eq. 3.1. Let the ergodic cost associated with the controlled diffusion $X$ and the running cost $r$ be defined as

[TABLE]

• (P1′)**

(unconstrained problem) The running cost function $r(x,u)$ is as in Eq. 3.4 with $\zeta>0$. The ergodic control problem is then defined as

[TABLE]

• (P2′)**

(constrained problem) The running cost function $r_{\mathsf{o}}(x,u)$ is as in Eq. 3.4 with $\zeta\equiv 0$. Also define

[TABLE]

with $\tilde{m}\geq 1$, and let $\updelta=(\updelta_{1},\dotsc,\updelta_{J})$ be a positive vector. The ergodic control problem under idleness constraints is defined as

[TABLE]

The quantities $\varrho^{*}(x)$ and $\varrho^{*}_{\mathsf{c}}(x)$ are called the optimal values of the ergodic control problems (P1*′) and (P2′*), respectively, for the controlled diffusion process $X$ with initial state $x$. Note that as is shown in Section 3 of [14] and Sections 3 and 5.4 of [15], the optimal values $\varrho^{*}(x)$ and $\varrho^{*}_{\mathsf{c}}(x)$ do not depend on $x\in\mathds{R}^{I}$, and thus we remove this dependence in the results stated in Section 3.3.

Let ${\mathscr{G}}$ denote the set of ergodic occupation measures corresponding to controls in $\mathfrak{U}_{\mathrm{SSM}}$, that is,

[TABLE]

where $\mathcal{L}^{u}f(x)$ is the controlled extended generator of the diffusion $X$ given in Eq. 2.25. The restriction of the ergodic control problem with running cost $r$ to stable stationary Markov controls is equivalent to minimizing $\uppi(r)\;=\;\int_{\mathds{R}^{I}\times{\mathbb{U}}}r(x,u)\,\uppi(\mathrm{d}{x},\mathrm{d}{u})$ over all $\uppi\in{\mathscr{G}}$. If the infimum is attained in ${\mathscr{G}}$, then we say that the ergodic control problem is well posed, and we refer to any $\bar{\uppi}\in{\mathscr{G}}$ that attains this infimum as an optimal ergodic occupation measure.

The characterization of the optimal solutions to the ergodic control problems (P1*′)–(P2′*) has been thoroughly studied in [14] and [15]. We refer the reader to these papers for relevant results used in the proof of asymptotic optimality which follows in the next section.

3.3. Asymptotic optimality results

We summarize here the main results on asymptotic optimality, which assert that the values of the two ergodic control problems in the diffusion scale converge to the values of the corresponding ergodic control problems for the limiting diffusion, respectively. The proofs of the asymptotic optimality are given in Section 7.

Recall the definitions of $J$, $J_{\mathsf{o}}$, $\hat{V}^{n}$, and $\hat{V}^{n}_{\mathsf{c}}$ in (P1)–(P2), and the definitions of $\varrho^{*}$ and $\varrho^{*}_{\mathsf{c}}$ in (P1*′)–(P2′*).

Theorem 3.1** (unconstrained problem).**

Suppose that $\gamma_{i}>0$ for some $i\in{\mathcal{I}}$. Then the following are true.

• (i)

$($lower bound$)$* For any sequence $\{Z^{n},\;n\in\mathds{N}\}\subset\bm{{\mathfrak{Z}}}$, the diffusion-scaled cost in Eq. 3.2 satisfies*

[TABLE]

• (ii)

$($upper bound$)$* $\displaystyle\limsup_{n\to\infty}\;\hat{V}^{n}(\hat{X}^{n}(0))\;\leq\;\varrho^{*}$ .*

Theorem 3.2** (constrained problem).**

Under the assumptions of Theorem 3.1, we have the following:

• (i)

$($lower bound$)$* Suppose that under a sequence $\{Z^{n},\;n\in\mathds{N}\}\subset\bm{{\mathfrak{Z}}}$, the constraint in Eq. 3.3 is satisfied for all sufficiently large $n\in\mathds{N}$. Then*

[TABLE]

and as a result we have that $~{}\displaystyle\liminf_{n\to\infty}\;\hat{V}^{n}_{\mathsf{c}}(\hat{X}^{n}(0))\,\geq\,\varrho_{\mathsf{c}}^{*}\,.$

• (ii)

$($upper bound$)$* For any $\epsilon>0$, there exists a sequence $\{Z^{n},\;n\in\mathds{N}\}\subset\bm{{\mathfrak{Z}}}$ such that the constraint in Eq. 3.3 is feasible for all sufficiently large $n$, and*

[TABLE]

Consequently, we have that $~{}\displaystyle\limsup_{n\to\infty}\;\hat{V}^{n}_{\mathsf{c}}(\hat{X}^{n}(0))\;\leq\;\varrho_{\mathsf{c}}^{*}$ .

4. BQBS Stability and Fairness

4.1. BQBS stable networks

It follows by Eq. 2.29 that the controlled diffusion limit for multiclass multi-pool networks have the following property. If under some admissible control (admissible scheduling policy) the mean empirical value of some power $\kappa\geq 1$ of the queueing and idleness processes is bounded, then the corresponding mean empirical value of the state process also remains bounded. This property also holds for the diffusion-scaled processes in the $n^{\text{th}}$ system, as shown later in Proposition 6.1.

There is however a large class of networks that share a more specific property, namely that the average value of any moment of a state process, is controlled by the average value of the corresponding moment of the queueing process alone. More precisely, the limiting diffusion of this class of networks satisfies

[TABLE]

for any $\kappa\geq 1$, and for a constant $C$ which depends only on $\kappa$. We refer to the class of networks which satisfy Eq. 4.1 as bounded-queue, bounded-state (BQBS) stable.

Define

[TABLE]

It follows by the proof of [14, Theorem 3.1] that a sufficient condition for BQBS stability is that Eq. 2.28 holds with ${\mathcal{K}}_{\delta}$ replaced by ${\mathcal{K}}_{\delta,+}$, i.e.,

[TABLE]

As shown later in Proposition 6.2, the inequality in Eq. 4.3 is sufficient for Eq. 4.1 to hold for the $n^{\text{th}}$ system, uniformly in $n\in\mathds{N}$. The class of networks which satisfy Eq. 4.3, and are therefore BQBS stable, includes the following special classes:

1. (i)

Networks with a single dominant class: there is only one class of jobs that can be served by more than one server pools (see Corollary 4.2 in [15]). This includes the standard “N” and “W” networks, the generalized “N” and “W” networks, and more general networks as depicted in Figure 1.

2. (ii)

Networks with the following parameter assumptions:

[TABLE]

for sufficiently small $\tilde{\delta}>0$. This includes networks with pool-dependent service rates, i.e., $\mu_{ij}=\bar{\mu}_{j}$ for all $(i,j)\in{\mathcal{E}}$, as a special class. (See Corollary 4.1 in [15]).

Remark 4.1*.*

For networks that satisfy Eq. 4.3, ergodic control problems with a running cost penalizing only the queue are well posed (e.g., we may allow $\zeta=0$ in Eq. 3.1). This is because, in the diffusion scale, the average value of the state process is controlled by the average value of the queue, and also by the fact, as shown in Lemma 2.2, that idleness is upper bounded by some multiple of the state.

4.2. The fairness problem

In addition to ergodic control problems as in (P1)–(P2), for BQBS stable networks we can also consider constrained problems which aim at balancing idleness among the server pools, and result in a fair allocation of idle servers. Let

[TABLE]

For the $n^{\rm th}$ system, we formulate this type of ergodic control problems as follows.

(P3) (fairness) Here we minimize the queueing cost while keeping the average idleness of the server pools balanced. Let $\uptheta=(\uptheta_{1},\dotsc,\uptheta_{J})^{\mathsf{T}}\in{\mathcal{S}}^{J}$ be a positive vector and let $1\leq\tilde{m}<m$. Let $\bar{J}_{\mathsf{c}}\coloneqq\sum_{\jmath\in{\mathcal{J}}}J_{\mathsf{c},\jmath}$. The associated cost minimization problem becomes

[TABLE]

For the corresponding diffusion, we have the following cost minimization problem.

(P3′)** (fairness) The running costs $r_{\mathsf{o}}$, and $r_{j}$, $j\in{\mathcal{J}}$, are as in (P2*′*). Let $\uptheta=(\uptheta_{1},\dotsc,\uptheta_{J})^{\mathsf{T}}\in{\mathcal{S}}^{J}$ be a positive vector, and $1\leq\tilde{m}<m$. The ergodic control problem under idleness fairness is defined as

[TABLE]

We next state an optimality result for the fairness problem (P3**′**). We first introduce some notation. Let

[TABLE]

For $\uptheta=(\uptheta_{1},\dotsc,\uptheta_{J})^{\mathsf{T}}\in\mathds{R}_{+}^{J}$ and $\uplambda=(\uplambda_{1},\dotsc,\uplambda_{J})^{\mathsf{T}}\in\mathds{R}^{J}_{+}$, define the running cost $h_{\uptheta,\uplambda}$ by

[TABLE]

where $\bar{r}=r_{1}+\dotsb+r_{J}$. We also let

[TABLE]

The following theorem characterizes of the optimal solution of (P3*′*)—see Theorem 5.8 in [15] and Theorem 4.3 in [16]. The existence of solutions to the HJB equation is proved for the diffusion control problem of the “N” network, but the argument used in the proof is applicable to the general multiclass multi-pool model discussed here. The uniqueness of the solutions $V_{\mathsf{f}}$ follows exactly as in the proof of Theorem 3.2 in [15].

Theorem 4.1**.**

Suppose that the network is BQBS stable and $\gamma_{i}>0$ for some $i\in{\mathcal{I}}$. Then the constraint in Eq. 4.4 is feasible for any positive vector $\uptheta=(\uptheta_{1},\dotsc,\uptheta_{J})^{\mathsf{T}}\in{\mathcal{S}}^{J}$. In addition, the following hold.

• (a)

There exists $\uplambda^{*}\in\mathds{R}^{J}_{+}$ such that

[TABLE]

• (b)

If $\uppi^{*}\in{\mathscr{H}}(\uptheta)$ attains the infimum of $\uppi\mapsto\uppi(r_{\mathsf{o}})$ in ${\mathscr{H}}(\uptheta)$, then $\uppi^{*}(r_{\mathsf{o}})\;=\;\uppi^{*}(h_{\uptheta,\uplambda^{*}})$, and

[TABLE]

• (c)

There exists $V_{\mathsf{f}}\in\mathcal{C}^{2}(\mathds{R}^{I})$ satisfying

[TABLE]

• (d)

A stationary Markov control $v_{\mathsf{f}}\in\mathfrak{U}_{\mathrm{SSM}}$ is optimal if and only if it satisfies

[TABLE]

where $H_{h_{\uptheta,\uplambda^{*}}}$ is defined in Eq. 4.5 with $r$ replaced by $h_{\uptheta,\uplambda^{*}}$.

• (e)

The map $\uptheta\mapsto\inf_{\uppi\,\in\,{\mathscr{H}}(\uptheta)}\;\uppi(r_{\mathsf{o}})$ is continuous at any feasible point $\hat{\uptheta}$.

We next state the asymptotic optimality result for this class of networks.

Theorem 4.2**.**

For the class of networks which satisfy Eq. 4.3, and $\gamma_{i}>0$ for some $i\in{\mathcal{I}}$, the following hold.

• (i)

$($lower bound$)$* there exists a positive constant $C_{\mathsf{f}}$ such that if a sequence $\{Z^{n},\;n\in\mathds{N}\}\subset\bm{{\mathfrak{Z}}}$ satisfies*

[TABLE]

for some $\epsilon>0$ and for all sufficiently large $n\in\mathds{N}$, then

[TABLE]

• (ii)

$($upper bound$)$* for any $\epsilon>0$, there exists a sequence $\{Z^{n},\;n\in\mathds{N}\}\subset\bm{{\mathfrak{Z}}}$ such that Eq. 4.6 holds for all sufficiently large $n\in\mathds{N}$, and*

[TABLE]

Remark 4.2*.*

The reader will certainly notice that whereas Theorem 4.1 holds for BQBS stable networks in general, i.e., networks which satisfy Eq. 4.1, the Foster–Lyapunov condition Eq. 4.3 is assumed in Theorem 4.2 which asserts asymptotic optimality. The reason behind this, is that the corresponding BQBS stability property should hold for the diffusion-scaled processes in order to establish the lower bound, and Eq. 4.3 needs to be invoked in order to assert this property (see Proposition 6.2 in Section 6). However, Eq. 4.3 is quite natural for the models considered here.

5. A family of stabilizing policies

We introduce a class of stationary Markov scheduling policies for the general multiclass multi-pool networks which is stabilizing for the diffusion-scaled state processes in the H–W regime. Let ${\mathcal{I}}_{\circ}\coloneqq\{i\in{\mathcal{I}}\colon\gamma_{i}=0\}$. Throughout this section we fix a collection $\{N_{ij}^{n}\in\mathds{N}\,,\;(i,j)\in{\mathcal{E}}\,,n\in\mathds{N}\}$ which satisfies

[TABLE]

We also define $\bar{N}_{i}^{n}\coloneqq\sum_{j\in{\mathcal{I}}(i)}N_{ij}^{n}$ for $i\in{\mathcal{I}}$.

Lemma 5.1**.**

Suppose that ${\mathcal{I}}_{\circ}\neq{\mathcal{I}}$. Then, given $\tilde{C}_{0}>0$, there exist a collection

[TABLE]

and a positive constant $\hat{C}_{0}$ satisfying

[TABLE]

and

[TABLE]

for all sufficiently large $n\in\mathds{N}$.

Proof.

Suppose, without loss of generality (WLOG), that ${\mathcal{I}}_{\circ}=\{1,\dotsc,I-1\}$. We claim that there exists a collection $\{\psi_{ij}\in\mathds{R}\colon j\in{\mathcal{J}}(i)\,,\;i\in{\mathcal{I}}\}$ of real numbers and a constant $C_{0}>0$, satisfying $\sum_{j\in{\mathcal{J}}(i)}\mu_{ij}\psi_{ij}>C_{0}$ for all $i\in{\mathcal{I}}_{\circ}$, and

[TABLE]

To prove the claim we use the argument of contradiction. If not, then by [38, Theorem 21.1] there exists a collection of nonnegative real numbers $\varkappa_{i}$, $i=1,\dotsc,{\mathcal{I}}$, such that

[TABLE]

for all $\{\psi_{ij}\}$ satisfying Eq. 5.4, and $\varkappa_{\hat{\imath}}>0$ for some $\hat{\imath}\in{\mathcal{I}}_{\circ}$. Since ${\mathcal{G}}$ is a tree, there exists a pair of finite sequences $i_{1},\dotsc,i_{\ell}$ and $j_{1},\dotsc,j_{\ell-1}$ such that $\hat{\imath}=i_{1}$, $i_{\ell}=I$, and $i_{k}\sim j_{k}$, $i_{k+1}\sim j_{k}$ for $k=1,\dotsc,\ell-1$. Choosing $\psi_{i_{\ell}j_{\ell-1}}=-1$, $\psi_{i_{\ell-1}j_{\ell-1}}=1$, $\psi_{ij_{\ell-1}}=0$ if $i\notin\{i_{\ell-1},i_{\ell}\}$, and $\psi_{ij}=0$ if $j\neq j_{\ell-1}$, it follows from Eq. 5.5 that $\varkappa_{i_{\ell-1}}=0$. Thus, WLOG, we may suppose that $\varkappa_{i_{2}}=0$. But then replacing $\psi_{i_{1},j_{1}}$ with $\psi_{i_{1},j_{1}}+C$, and $\psi_{i_{2},j_{1}}$ with $\psi_{i_{2},j_{1}}-C$, the new set of numbers $\{\psi_{ij}\}$ satisfies Eq. 5.4. Therefore, by Eq. 5.5 we must have

[TABLE]

for all $C\in\mathds{R}$, which is impossible since $\varkappa_{i_{1}}\mu_{i_{1}j_{1}}>0$. This proves the claim.

Scaling $\{\psi_{ij}\}$ by multiplying with a constant, we may assume that

[TABLE]

For each $j\in{\mathcal{J}}$, if ${\mathcal{I}}(j)$ is a singleton, i.e., ${\mathcal{I}}(j)=\{i_{1}\}$, then we define $\tilde{N}_{i_{1}j}^{n}\coloneqq N_{i_{1}j}^{n}$. Otherwise, if ${\mathcal{I}}(j)=\{i_{1},\dotsc,i_{\ell}\}$, then we let \tilde{N}_{i_{k}j}^{n}\coloneqq N_{i_{k}j}^{n}+\bigl{\lfloor}\psi_{i_{k}j}\sqrt{n}\bigr{\rfloor} for $k=1,\dotsc,\ell-1$, and $\tilde{N}_{i_{\ell}j}^{n}\coloneqq N_{j}^{n}-\sum_{k=1}^{\ell-1}\tilde{N}_{i_{k}j}^{n}$. It is clear then that Eq. 5.1 holds for all sufficiently large $n$ by Eq. 5.6, while Eqs. 5.2 and 5.3 hold by construction. This completes the proof. ∎

Definition 5.1**.**

Let $\{\tilde{N}_{ij}^{n}\}$ be as in Lemma 5.1, and $\tilde{N}_{i}^{n}\coloneqq\sum_{j\in{\mathcal{I}}(i)}\tilde{N}_{ij}^{n}$ for $i\in{\mathcal{I}}$. Let ${\mathscr{Z}}^{n}$ denote the class of Markov policies $z$ satisfying

[TABLE]

We refer to this class of Markov policies as balanced saturation policies (BSPs).

We remark that if all $\gamma_{i}>0$ for $i\in{\mathcal{I}}$, then in Definition 5.1, we may replace $\tilde{N}_{ij}^{n}$ and $\tilde{N}^{n}_{i}$ by $N^{n}_{ij}$ and $\bar{N}^{n}_{i}$, respectively. Note that by Lemma 5.1, the quantities $\tilde{N}^{n}_{ij}$ and $\tilde{N}_{i}$ are within ${\mathscr{O}}(\sqrt{n})$ of the quantities $N^{n}_{ij}$ and $\bar{N}_{i}$, which can be regarded as the ‘steady-state’ allocations for the $n^{\text{th}}$ system. Thus, in the class of BSPs, if $\gamma_{i}>0$ for some $i$, then the scheduling policy $z$ is determined using the ‘shifted’ steady-state allocations $\tilde{N}^{n}_{ij}$ and $\tilde{N}_{i}$.

Note that the stabilizing policy for the ‘N’ network in [15] belongs to the class of BSPs. As another example, for the ‘M’ network, if $\gamma_{i}>0$ for some $i=1,2$, the scheduling policy $z=z(x)$, $x\in\mathds{Z}^{2}_{+}$, defined by

[TABLE]

is a BSP. If $\gamma_{i}>0$ for $i=1,2$, then in the scheduling policy above, we can replace $\tilde{N}^{n}_{ij}$ and $\tilde{N}_{i}^{n}$ by $N^{n}_{ij}$ and $\bar{N}_{i}^{n}$, respectively.

Using the function $\hat{x}^{n}$ in Definition 2.3, we can write the generator $\widehat{\mathscr{L}}_{n}^{z}$ of the diffusion-scaled state process $\hat{X}^{n}$ under the policy $z\in{\mathscr{Z}}^{n}$ as

[TABLE]

where ${\mathscr{L}}_{n}^{z}$ is as defined in Eq. 2.7.

Recall that a $\mathds{R}^{d}$-valued Markov process $\{M_{t}\colon t\geq 0\}$ is called exponentially ergodic if it possesses an invariant probability measure $\overline{\uppi}(\mathrm{d}y)$ satisfying

[TABLE]

for some $\kappa>0$, where $P_{t}(x,\cdot)\coloneqq\mathbb{P}^{x}(M_{t}\in\cdot)$ denotes the transition probability of $M_{t}$, and $\lVert\cdot\rVert_{\mathrm{TV}}$ denotes the total variation norm.

Proposition 5.1**.**

Let $\widehat{\mathscr{L}}_{n}^{z}$ denote the generator of the diffusion-scaled state process $\hat{X}^{n}$ under a BSP $z\in{\mathscr{Z}}^{n}$. Let $\widetilde{\mathcal{V}}_{\epsilon,\beta}$ be as in Definition 2.6, with $\beta\in\mathds{R}^{I}$ a positive vector. There exists $\epsilon>0$, and positive constants $C_{0}$ and $C_{1}$ such that

[TABLE]

The process $\hat{X}^{n}$ is exponentially ergodic and admits a unique invariant probability measure $\hat{\uppi}^{n}$ satisfying

[TABLE]

for any $\kappa<C_{1}$, where $P^{n}_{t}(x,\cdot)$ denotes the transition probability of $\hat{X}^{n}$.

Proof.

Throughout the proof we use, without further mention, the fact that there exists a constant $\tilde{C}_{0}$ such that

[TABLE]

for all $i\in{\mathcal{I}}$ and all sufficiently large $n\in\mathds{N}$. This follows by Eq. 2.1.

Recall the collection $\{\tilde{N}_{ij}^{n}\in\mathds{N}\,,\;(i,j)\in{\mathcal{E}}\}$ constructed in Lemma 5.1 with respect to the constant $\tilde{C}_{0}$ given above. Define $\breve{x}_{i}=\breve{x}_{i}^{n}(x)\coloneqq x_{i}-\tilde{N}_{i}^{n}$, $\hat{\vphantom{\rule{1.0pt}{5.93056pt}}\smash{\hat{x}}}_{i}\coloneqq\frac{1}{\sqrt{n}}\breve{x}_{i}^{n}$, and let

[TABLE]

Using the identity

[TABLE]

we obtain

[TABLE]

for some constant $\tilde{\kappa}_{1}>0$, and all $\epsilon\in(0,1)$, with

[TABLE]

Fix $n\in\mathds{N}$. By Eq. 2.7, with

[TABLE]

for $i\in{\mathcal{I}}$ (see Eq. 2.14), and using Eq. 5.11, we obtain

[TABLE]

where

[TABLE]

It always holds that $z_{ij}\leq x_{i}$ and $q_{i}\leq x_{i}$ for all $(i,j)\in{\mathcal{E}}$. By Eq. 2.1, for some constant $\tilde{\kappa}_{2}$ we have

[TABLE]

and all $n\in\mathds{N}$. Thus, by Eqs. 5.13 and 5.14, we obtain

[TABLE]

We next calculate an estimate for $F_{n,i}^{(1)}$ in Eq. 5.13. Consider any $i\in{\mathcal{I}}$. Define

[TABLE]

We distinguish three cases.

Case A. Suppose that $x_{i}<\tilde{N}_{i}^{n}$. We write

[TABLE]

Note that $z_{ij}^{n}-\tilde{N}_{ij}^{n}\leq 0$ and $\breve{x}_{i}\leq 0$. Therefore, we have

[TABLE]

We also have that

[TABLE]

Thus we obtain

[TABLE]

Case B. Suppose that

[TABLE]

Define

[TABLE]

and note that $\breve{\zeta}_{ij}\geq 0$. Then $z_{ij}=\breve{z}_{ij}=\tilde{N}_{ij}^{n}+\breve{\zeta}_{ij}$.

Suppose first that $\gamma_{i}=0$. Then by Eq. 5.16, we have

[TABLE]

Suppose now that $\gamma_{i}>0$. Then by Eq. 5.16, we have

[TABLE]

Case C. Suppose that

[TABLE]

Let $\hat{\jmath}\in{\mathcal{J}}(i)$ be arbitrary. We have

[TABLE]

From cases A–C, we obtain

[TABLE]

It is clear from these estimates together with Sections 5 and 5, that, for $\varepsilon>0$ small enough, there exist positive constants $M_{k}$, $k=0,1$, satisfying

[TABLE]

Define $\varsigma^{n}_{i}\coloneqq\sqrt{n}x_{i}^{*}-\tfrac{1}{\sqrt{n}}\tilde{N}_{i}^{n}$. Then $\widehat{\mathcal{V}}_{\epsilon,\beta}(x)=\widetilde{\mathcal{V}}_{\epsilon,\beta}(\hat{x}+\varsigma^{n})$. Thus, using the definition in Eqs. 5.7 and 2.3, we obtain

[TABLE]

Since $\lvert\varsigma^{n}_{i}\rvert\leq\hat{C}_{0}$ by Lemma 5.1, it is clear that Eq. 5.8 follows by Eq. 5.17.

It is standard to show that Eq. 5.8 implies Eq. 5.9. One can apply, for example, equation (3.5) in [39, Theorem 3.2], using $\varPsi_{1}(x)=x$ and $\varPsi_{2}(x)=1$. This completes the proof. ∎

The following is immediate from Proposition 5.1.

Corollary 5.1**.**

If $z\in{\mathscr{Z}}^{n}$ is a BSP, then for some $\varepsilon>0$ we have

[TABLE]

and the same holds if we replace $\hat{X}^{n}$ with $\hat{Q}^{n}$ or $\hat{Y}^{n}$ in Eq. 5.18. In particular, the invariant probability measure of the diffusion-scaled process $\hat{X}^{n}(t)$ under a BSP has finite moments of any order.

Remark 5.1*.*

A Foster–Lyapunov property analogous to Eq. 2.30 can be obtained for the diffusion-scaled state process $\hat{X}^{n}$ under a BSP policy. Let ${\mathcal{V}}_{\kappa,\beta}$ be as in Definition 2.6 and $\widehat{\mathscr{L}}_{n}^{z}$ as in Proposition 5.1. One can show, by a slight modification of the proof of Proposition 5.1, that for each $\kappa>1$, there exist positive constants $C_{0}$ and $C_{1}$ depending only on $\kappa$ and $n_{0}\in\mathds{N}$, such that for all $z\in{\mathscr{Z}}^{n}$, we have

1. (i)

If $\gamma_{i}>0$ for all $i\in{\mathcal{I}}$, then

[TABLE] 2. (ii)

If $\gamma_{i}>0$ for some $i\in{\mathcal{I}}$ (but not all $i$), then

[TABLE]

The Foster–Lyapunov property in Eq. 5.19 can be equivalently written as

[TABLE]

for some constant $C_{1}^{\prime}$. Such Foster–Lyapunov properties appear in the studies on subexponential rate of convergence of Markov processes (see, e.g., [39], [40] and references therein). Thus Eq. 5.19 provides an interesting example to that rich theory. On the other hand, Eq. 5.8 with the exponential function $\widetilde{\mathcal{V}}_{\epsilon,\beta}$ is stronger, and implies exponential ergodicity of the processes $\hat{X}^{n}(t)$ under a BSP.

6. Ergodic Properties of the $n^{\mathrm{th}}$ System

6.1. Moment bounds for general multiclass multi-pool networks

Recall the moment bounds for the diffusion limit in Eq. 2.29. We prove the analogous property for the diffusion-scaled state process $\hat{X}^{n}$.

Proposition 6.1**.**

For any $\kappa\geq 1$, there exist constants $\widetilde{C}_{0}$ and $\widetilde{C}_{1}$, depending only on $\kappa$, such that

[TABLE]

for all $n\in\mathds{N}$, and for any sequence $\{Z^{n}\in{\mathfrak{Z}}^{n},\;n\in\mathds{N}\}$.

Proof.

Let $\mathscr{V}(x)\;\coloneqq\;\sum_{i\in{\mathcal{I}}}\beta_{i}\mathscr{V}_{i}(x_{i})$, $x\in\mathds{R}^{I}$, where $\beta_{i}$, $i\in{\mathcal{I}}$, are positive constants to be determined later, and $\mathscr{V}_{i}(x_{i})=\lvert x_{i}\rvert^{\kappa}$ when $\kappa>1$, whereas $\mathscr{V}_{i}(x_{i})=\frac{\lvert x_{i}\rvert^{2}}{\sqrt{\delta+\lvert x_{i}\rvert^{2}}}$ for some $\delta>0$ when $\kappa=1$. By applying Itô’s formula on $\mathscr{V}$, we obtain from Eq. 2.12 that for $t\geq 0$,

[TABLE]

where $\mathscr{A}^{n}$ is given in Definition 2.4, and $\mathscr{D}\mathscr{V}(\hat{X}^{n},s)$ is as in Eq. 2.20.

Let $\hat{\Theta}^{n}\coloneqq e\cdot\hat{Q}^{n}\wedge e\cdot\hat{Y}^{n}$. Then \hat{Q}^{n}=\bigl{(}\hat{\Theta}^{n}+(e\cdot\hat{X}^{n})^{+}\bigr{)}\hat{u}^{c} and \hat{Y}^{n}=\bigl{(}\hat{\Theta}^{n}+(e\cdot\hat{X}^{n})^{-}\bigr{)}\hat{u}^{s} for some $(\hat{u}^{c},\hat{u}^{s})\in{\mathbb{U}}$ by Eq. 2.13. Also by the linearity of the map $\Psi$ in Eq. 2.9, we obtain

[TABLE]

Define

[TABLE]

and

[TABLE]

By the convergence of the parameters in Eq. 2.1, we have

[TABLE]

for all $i\in{\mathcal{I}}$, for some constant $c_{1}(n)\searrow 0$ as $n\to\infty$. Therefore

[TABLE]

for some constant $c^{\prime}_{1}(n)\searrow 0$ as $n\to\infty$.

Recall the drift representation $b(x,u)$ in Eq. 2.26. By Section 6.1, we obtain that for each $i$,

[TABLE]

Since $-B_{1}$ in Eq. 2.26 is lower diagonal and Hurwitz, there exist positive constants $\beta_{i}$, $i\in{\mathcal{I}}$, such that

[TABLE]

for some positive constant $c_{2}$. Thus, applying Young’s inequality, after some simple calculations, we obtain

[TABLE]

for some positive constants $c_{3}$ and $c_{4}$.

Concerning the last term in Eq. 6.2, we first note that by the definition of $\mathscr{V}_{i}$, since the jump size is of order $\frac{1}{\sqrt{n}}$, there exists a positive constant $c_{5}$ such that $\sup_{|x^{\prime}_{i}-x_{i}|\leq 1}\lvert\mathscr{V}_{i}^{\prime\prime}(x^{\prime}_{i})\rvert\leq c_{5}(1+|x_{i}|^{\kappa-2})$ for each $x_{i}\in\mathds{R}$. Then by the Taylor remainder theorem, we obtain that for each $i\in{\mathcal{I}}$,

[TABLE]

Thus, we have

[TABLE]

for some positive constant $c_{6}$, independent of $n$, where in the first equality we use the fact that the optional martingale $[\hat{X}^{n}_{i}]$ is the sum of the squares of the jumps, and that $[\hat{X}^{n}_{i}]-\langle\hat{X}^{n}_{i}\rangle$ is a martingale, and in the last inequality we use Young’s inequality.

Therefore, the assertion of the proposition follows by combining Eqs. 6.2, 6.5, 6.7 and 6.1, and the inequality

[TABLE]

6.2. Moment bounds for BQBS stable networks

For the class of BQBS stable networks, the moment bound in Eq. 4.1 holds for the limiting diffusion $X$. The following proposition shows that the analogous moment bound also holds for the diffusion-scaled process $\hat{X}^{n}$ of this class of networks.

Proposition 6.2**.**

Suppose that Eq. 4.3 holds. Then Proposition 6.1 holds with Eq. 6.1 replaced by

[TABLE]

for all $n\in\mathds{N}$.

Proof.

Recall the definition of the cone ${\mathcal{K}}_{\delta,+}$ in Eq. 4.2. By Eqs. 4.3, 6.4 and 6.6, we obtain

[TABLE]

for some positive constants $c_{0}$ and $c_{1}$. Since

[TABLE]

we obtain by Section 6.2 that

[TABLE]

for some positive constants $c_{i}^{\prime}$, $i=0,1$, where the second inequality in Section 6.2 follows by applying Young’s inequality. The rest follows as in the proof of Proposition 6.1. ∎

As a consequence of Proposition 6.2, we also obtain the following moment bound for the idleness process.

Corollary 6.1**.**

If Eq. 4.3 holds, then there exist some constants $\widetilde{C}_{0}^{\prime}>0$ and $\widetilde{C}_{1}^{\prime}>0$ such that

[TABLE]

for all $n\in\mathds{N}$, and for any sequence $\{Z^{n}\in{\mathfrak{Z}}^{n},\;n\in\mathds{N}\}$.

Proof.

The claim follows from Eq. 6.9 and the fundamental identities $\lVert\hat{Q}^{n}\rVert=\hat{\Theta}^{n}+(e\cdot\hat{X}^{n})^{+}$ and $\lVert\hat{Y}^{n}\rVert=\hat{\Theta}^{n}+(e\cdot\hat{X}^{n})^{-}$. ∎

6.3. Convergence of mean empirical measures

For the process $\hat{X}^{n}$ under a scheduling policy $Z^{n}$, and with $U^{n}$ as in Definition 2.5, we define the mean empirical measures

[TABLE]

for Borel sets $A\subset\mathds{R}^{I}$ and $B\subset{\mathbb{U}}$. Recall Definition 2.2. The lemma which follows shows that if the long-run average first-order moment of the diffusion-scaled state process under an EJWC scheduling policy is finite, then the mean empirical measures $\Phi^{Z^{n}}_{T}$ are tight and converge to an ergodic occupation measure corresponding to some stationary stable Markov control for the limiting diffusion control problem. This property is used in the proof of the lower bounds in Theorems 3.1 and 3.2.

Lemma 6.1**.**

If under some sequence of scheduling policies $\{Z^{n},\;n\in\mathds{N}\}\subset\bm{{\mathfrak{Z}}}$, we have

[TABLE]

then \bigl{\{}\Phi^{Z^{n}}_{T}\colon n\in\mathds{N},\ T>0\bigr{\}} is tight, and any limit point $\uppi\in{\mathcal{P}}(\mathds{R}^{I}\times{\mathbb{U}})$ of \bigl{\{}\Phi^{Z^{n}}_{T}\bigr{\}} over a sequence $(n_{k},T_{k})$, with $n_{k}\to\infty$ and $T_{k}\to\infty$, lies in ${\mathscr{G}}$.

Proof.

It is clear that Eq. 6.13 implies that \bigl{\{}\Phi^{Z^{n}}_{T}\colon n\in\mathds{N},\ T>0\bigr{\}} is tight. For $f\in\mathcal{C}^{\infty}_{c}(\mathds{R}^{I})$, by Eq. 2.19, the definition of $\Phi^{Z^{n}}_{T}$ in Eq. 6.12, and Lemma 2.3, we obtain

[TABLE]

Let

[TABLE]

By Taylor’s formula, using also the fact that the jump size is $\frac{1}{\sqrt{n}}$, we obtain

[TABLE]

for some constant $\kappa$ that does not depend on $n\in\mathds{N}$. On the other hand, since independent Poisson processes have no simultaneous jumps w.p.1., we have

[TABLE]

and the right hand side is uniformly bounded over $n\in\mathds{N}$ and $T>0$ by Eq. 6.13.

Therefore, taking limits in Eq. 6.14, we obtain

[TABLE]

Let $(n_{k},T_{k})$, with $n_{k}\to\infty$ and $T_{k}\to\infty$, be any sequence along which $\Phi^{Z^{n}}_{T}(\mathrm{d}{x},\mathrm{d}{u})$ converges to some $\uppi\in{\mathcal{P}}(\mathds{R}^{I}\times{\mathbb{U}})$. By Eq. 6.15, Lemma 2.5, and a standard triangle inequality, we obtain

[TABLE]

This implies that $\uppi\in{\mathscr{G}}$. ∎

We introduce a canonical construction of scheduling policies which is used in the proofs of the upper bounds for asymptotic optimality. Recall Definitions 2.2 and 2.3, and $\breve{{\mathscr{X}}}^{n}$ defined in Eq. 2.10.

Definition 6.1**.**

Let $\varpi\colon\{x\in\mathds{R}^{I}_{+}\colon e\cdot x\in\mathbb{Z}\}\to\mathbb{Z}^{I}_{+}$ be a measurable map defined by

[TABLE]

By abuse of notation, we denote by $\varpi$ the similarly defined map $\varpi\colon\{x\in\mathds{R}^{J}_{+}\colon e\cdot x\in\mathbb{Z}\}\to\mathbb{Z}^{J}_{+}$. For a precise control $v\in\mathfrak{U}_{\mathrm{SSM}}$, define the maps $q^{n}[v]\colon\mathds{R}^{I}\to\mathbb{Z}^{I}_{+}$ and $y^{n}[v]\colon\mathds{R}^{I}\to\mathbb{Z}^{J}_{+}$ by

[TABLE]

where $\hat{x}\in{\mathscr{S}}^{n}$. Recall the definition of the linear map $\Psi$ in Eq. 2.9. Define the Markov scheduling policy $z^{n}[v]$ on $\breve{{\mathscr{S}}}^{n}$ by

[TABLE]

Corollary 6.2**.**

For any precise control $v\in\mathfrak{U}_{\mathrm{SSM}}$, we have

[TABLE]

for all $x\in\breve{{\mathscr{X}}}^{n}$, i.e., the JWC condition is satisfied in $\breve{{\mathscr{X}}}^{n}$.

Proof.

This follows from Lemma 2.1 and the definition of the maps $q^{n}[v]$, $y^{n}[v]$ and $z^{n}[v]$. ∎

The next lemma is used in the proof of upper bounds in Theorems 3.1 and 3.2. It shows that for any given continuous precise stationary stable Markov control, if we construct a sequence of EJWC scheduling policies as in Definition 6.1, then the corresponding mean empirical measures of the diffusion-scaled processes converge and the limit agrees with the ergodic occupation measure of the limiting diffusion corresponding to that control.

Lemma 6.2**.**

Let $v\in\mathfrak{U}_{\mathrm{SSM}}$ be a continuous precise control, and \bigl{\{}Z^{n}\,\colon\,n\in\mathds{N}\bigr{\}} be any sequence of admissible scheduling policies such that each $Z^{n}$ agrees with the Markov scheduling policy $z^{n}[v]$ given in Definition 6.1 on $\sqrt{n}\breve{B}$, i.e., Z^{n}(t)=z^{n}[v]\bigl{(}\hat{X}^{n}(t)\bigr{)} whenever $\hat{X}^{n}(t)\in\sqrt{n}\breve{B}$. For $\hat{x}\in\sqrt{n}\breve{B}\cap{\mathscr{S}}^{n}$, we define

[TABLE]

For the process $X^{n}$ under the scheduling policy $Z^{n}$, define the mean empirical measures

[TABLE]

for Borel sets $A\subset\sqrt{n}\breve{B}$ and $B\subset{\mathbb{U}}$. Suppose that Eq. 6.13 holds under this sequence $\{Z^{n}\}$. Then the ergodic occupation measure $\uppi_{v}$ of the controlled diffusion in Eq. 2.24 corresponding to $v$ is the unique limit point in ${\mathcal{P}}(\mathds{R}^{d}\times{\mathbb{U}})$ of $\tilde{\Phi}^{Z^{n}}_{T}$ as $n$ and $T$ tend to $\infty$.

Proof.

The proof follows exactly as that of Lemma 7.2 in [16]. ∎

6.4. A stability preserving property in the JWC region

If there exists a stationary Markov control under which the controlled diffusion is exponentially ergodic, then it can be shown that under the corresponding scheduling policy as constructed in Definition 6.1, the diffusion-scaled state process satisfies a Foster–Lyapunov condition of the exponential ergodicity type in the JWC region. We refer to this as the stability preserving property in the JWC region. This property is important to prove the upper bounds for the asymptotic optimality, and is also the reason why the spatial truncation technique works. We present this in the following Proposition.

Proposition 6.3**.**

Let $\widetilde{{\mathcal{V}}}_{\epsilon,\beta}$ be as in Definition 2.6. Suppose $v\in\mathfrak{U}_{\mathrm{SSM}}$ is such that for some positive constants $c_{0}$, $c_{1}$, and $\epsilon>0$, and a positive vector $\beta\in\mathds{R}^{I}$, it holds that

[TABLE]

Let $\hat{X}^{n}$ denote the diffusion-scaled state process under the scheduling policy $z^{n}[v]$ in Definition 6.1, and $\widehat{\mathscr{L}}_{n}$ denote its generator. Then, there exists $n_{0}\in\mathds{N}$ such that

[TABLE]

for some positive constants $\hat{c}_{0}$ and $\hat{c}_{1}$, and for all $n\geq n_{0}$.

Proof.

Recall the notation $\hat{x}=\hat{x}^{n}(x)$ in Definition 2.3. Under the Markov scheduling policy $z^{n}[v]$ in Definition 6.1, for each given $x\in\mathds{R}^{I}$, we define the associated diffusion-scaled quantities

[TABLE]

Recall that $\widehat{{\mathscr{L}}}_{n}=\widehat{{\mathscr{L}}}_{n}^{z^{n}[v]}$ denotes the generator of $\hat{X}^{n}$ under the scheduling policy $z^{n}[v]$ (see Eqs. 2.7 and 5.7). Let $\widehat{\mathcal{V}}_{\epsilon,\beta}(x)\;\coloneqq\;\widetilde{\mathcal{V}}_{\epsilon,\beta}(\hat{x})$, with $\widetilde{\mathcal{V}}_{\epsilon,\beta}$ as in Definition 2.6. Using the identity in Eq. 5.10, we obtain

[TABLE]

for some constant $\hat{\kappa}_{1}>0$, and all $\epsilon\in(0,1)$. Thus by Eqs. 2.7, 5.7 and 6.18, we obtain

[TABLE]

in direct analogy to Section 5, where

[TABLE]

The dependence of $G_{n,i}^{(1)}$ and $G_{n,i}^{(2)}$ on $\hat{x}$ is implicit through $z^{n}[v]$. By Definition 6.1, it always holds that $z^{n}_{ij}\leq x_{i}$ and $q^{n}_{i}\leq x_{i}$ for all $(i,j)\in{\mathcal{E}}$. Since $\frac{x_{i}}{\sqrt{n}}=\hat{x}_{i}+\sqrt{n}x^{*}_{i}$, and $\hat{z}_{ij}=\frac{x_{i}}{\sqrt{n}}+z_{ij}^{*}$, we obtain

[TABLE]

for some constant $\hat{\kappa}_{2}>0$, where the last inequality follows from the assumption on the parameters in Eq. 2.1.

Since the control $v$ satisfies Eq. 6.17, we must have, for some positive constants $c_{0}^{\prime}$ and $c_{1}^{\prime}$ that

[TABLE]

By Lemmas 2.3 and 2.5, we have G_{n,i}^{(1)}(\hat{x})\to b_{i}\bigl{(}\hat{x},v(\hat{x})\bigr{)}, uniformly over compact sets of $\mathds{R}^{I}$ as $n\to\infty$. Therefore, the result follows by combining Eqs. 6.19, 6.4 and 6.21. ∎

7. Proofs of Asymptotic Optimality

We need the following lemma, which is used in the proof of the upper bound.

Lemma 7.1**.**

For any $\varepsilon>0$, there exists a continuous precise control $v_{\epsilon}\in\mathfrak{U}_{\mathrm{SSM}}$ with the following properties:

1. (a)

For some positive vector $\beta\in\mathds{R}^{I}$ which does not depend on $\varepsilon$, and any $\kappa>1$, we have

[TABLE]

for some constants $c_{0}$ and $c_{1}$ depending only on $\kappa$. 2. (b)

With $\uppi_{v_{\varepsilon}}$ denoting the ergodic occupation measure corresponding to $v_{\varepsilon}$, it holds that

[TABLE]

where $\varrho^{*}$ is the optimal value of problem (P1′).

Proof.

By [15, Theorem 4.2] there exists a constant Markov control $\bar{u}$ and a positive vector $\beta\in\mathds{R}^{I}$ satisfying

[TABLE]

for all $\kappa>1$ and some constants $\bar{c}_{0}$ and $\bar{c}_{1}$. Even though not stated in that theorem, it follows from its proof that the constants $\bar{c}_{0}$ and $\bar{c}_{1}$ depend only on $\kappa$. We perturb $r$ by adding a positive strictly convex function $f\colon{\mathbb{U}}\to\mathds{R}_{+}$, such that the optimal value of the problem (P1′) with $r$ replaced by $r+f$ is smaller than $\varrho^{*}+\frac{\varepsilon}{3}$. Following the proof of Theorems 4.1 and 4.2 in [14], there exists $R>0$ large enough and a stationary Markov control $\bar{v}_{R}$, which agrees with $\bar{u}$ on $B_{R}^{c}$ and satisfies $\uppi_{\bar{v}_{R}}(r+f)<\varrho^{*}+\frac{2\varepsilon}{3}$. This control satisfies, for some $V_{R}\in\mathcal{C}^{2}(B_{R})$,

[TABLE]

for all $x\in B_{R}$. Since $u\mapsto\{b(x,u)\cdot p+r(x,u)+f(u)\}$ is strictly convex whenever it is not constant, it follows by Eq. 7.3 that $\bar{v}_{R}$ is continuous on $B_{R}$. Consider the concatenated Markov control which agrees with $\bar{v}_{R}$ in $B_{R}$ and with $\bar{u}$ in $B_{R}^{c}$. As in the proof of [14, Theorem 2.2], we can employ a cut-off function to smoothen the discontinuity of this control at the concatenation boundary, and thus obtain a Markov control $v_{\varepsilon}$ satisfying $\uppi_{v_{\varepsilon}}(r+f)<\varrho^{*}+\varepsilon$. Clearly then part (b) holds since $f$ is nonnegative, while part (a) holds by Eq. 7.2 and the fact that $v_{\varepsilon}$ agrees with $\bar{u}$ outside a compact set. This completes the proof. ∎

Concerning the constrained problem (P2′) and the fairness problem (P3*′*) we have the following analogous result.

Corollary 7.1**.**

For any $\varepsilon>0$, there exists a continuous precise control $v_{\epsilon}\in\mathfrak{U}_{\mathrm{SSM}}$ satisfying Lemma 7.1 (a), and constants $\updelta^{\epsilon}_{j}<\updelta_{j}$, $j\in{\mathcal{J}}$ such that:

• (i)

In the case of problem (P2′) we have

[TABLE]

• (ii)

In the case of problem (P3′) we have

[TABLE]

Proof.

Part (i) follows as in Theorem 5.7 in [15]. The proof of part (ii) is completely analogous. ∎

7.1. Proof of Theorem 3.1

Proof of the lower bound.

Without loss of generality, suppose that $Z^{n_{k}}\in{\mathfrak{Z}}^{n_{k}}$, for some increasing sequence $\{n_{k}\}\subset\mathds{N}$, is a collection of scheduling policies in $\bm{{\mathfrak{Z}}}$ such that J\bigl{(}\hat{X}^{n_{k}}(0),Z^{n_{k}}\bigr{)} converges to a finite value as $k\to\infty$. Denote by $\Phi^{n_{k}}_{T}$, the mean empirical measure $\Phi^{Z^{n_{k}}}_{T}$ defined in Eq. 6.12. Then by Proposition 6.1 and the definitions of the running cost $\hat{r}$ in Eq. 3.1 and J\bigl{(}\hat{X}^{n}(0),Z^{n}\bigr{)} in Eq. 3.2 we obtain

[TABLE]

It is also clear by the definition of $\Phi^{n}_{T}$ and $J$ that we can select a sequence $\{T_{k}\}\subset\mathds{R}_{+}$, with $T_{k}\to\infty$, such that

[TABLE]

By Lemma 6.1 and Eq. 7.4, $\{\Phi^{n_{k}}_{T_{k}}\,\colon\,k\in\mathds{N}\}$ is tight and the limit of any converging subsequence $\{\Phi^{n_{k}^{\prime}}_{T_{k}^{\prime}}\}$ is in ${\mathscr{G}}$. Therefore it follows by Eq. 7.5 that

[TABLE]

and this completes the proof. ∎

Proof of the upper bound.

Recall ${\mathcal{V}}_{\kappa,\beta}$ in Definition 2.6. Let $\kappa=m+2$. By Lemma 7.1, there exists a continuous precise control $v_{\varepsilon}$ such that the corresponding ergodic occupation measure satisfies $\uppi_{v_{\varepsilon}}(r)\;<\;\varrho^{*}+\varepsilon$, and Eq. 7.1 holds.

For the $n^{\text{th}}$ system, we construct a concatenated Markov scheduling policy $\mathring{z}^{n}$ as follows. Recall Definition 2.2. Inside $\breve{{\mathscr{X}}}^{n}$, we apply the stationary policy $z^{n}[v_{\epsilon}]$ as in Definition 6.1, and outside $\breve{{\mathscr{X}}}^{n}$, we apply some Markov scheduling policy $z\in{\mathscr{Z}}^{n}$ in Definition 5.1 that is exponentially stable. By Propositions 5.1 and 6.3 there exist positive constants $\hat{c}_{0}$, $\hat{c}_{1}$, a positive vector $\beta\in\mathds{R}^{I}$, and $n_{0}\in\mathds{N}$, such that

[TABLE]

This immediately implies that $\sup_{n\geq n_{0}}\,J(\hat{X}^{n}(0),Z^{n})<\infty$. Let $\tilde{\Phi}^{n}_{T}\equiv\tilde{\Phi}^{\mathring{z}^{n}}_{T}$ be the corresponding mean empirical measures as defined in Eq. 6.16. Then the Foster–Lyapunov condition in Eq. 7.6 implies that we can choose a sequence $\{T_{n}\}$ such that

[TABLE]

WLOG, we assume that $T_{n}\to\infty$.

It is clear that $\mathring{z}^{n}$ can be viewed as a function of $\hat{x}\in{\mathscr{S}}^{n}$. We let $\hat{\mathring{z}}^{n}_{ij}(\hat{x})\;\coloneqq\;\frac{(\mathring{z}^{n}_{ij}(\hat{x})-nz^{*})}{\sqrt{n}}$ as in Definition 2.3. In analogy to Eqs. 2.15 and 2.16 we define

[TABLE]

The running cost $\hat{r}$ is uniformly integrable with respect to the collection $\{\tilde{\Phi}^{n}_{T}\,,\ n\in\mathds{N}\,,\,T\geq 0\}$ by Eq. 7.7. Thus by Birkhoff’s ergodic theorem, for any $\eta>0$, we can choose a ball $B(\eta)$, and a sequence $T_{n}$ such that

[TABLE]

for all $T\geq T_{n}$.

By the JWC condition on $\{\hat{x}\in\breve{\mathscr{S}}^{n}\}$ and Corollary 6.2, we have (e\cdot\hat{\mathring{q}}^{n}(\hat{x})\bigr{)}^{+}=(e\cdot\hat{x})^{+} and (e\cdot\hat{\mathring{y}}^{n}(\hat{x})\bigr{)}^{+}=(e\cdot\hat{x})^{-} for all $\hat{x}\in B(\eta)$, and for all large enough $n$. On the other hand we have

[TABLE]

Since $v_{\varepsilon}$ is a continuous precise control then $\tilde{\Phi}^{n}_{T}$ converges to $\uppi_{v_{\epsilon}}$ in ${\mathcal{P}}(\mathds{R}^{I}\times{\mathbb{U}})$ as $n$ and $T$ tend to $\infty$ by Lemma 6.2. Thus, using Eq. 7.9 and a triangle inequality, we obtain

[TABLE]

By Eqs. 7.8 and 7.10 we obtain

[TABLE]

Since $\eta$ and $\epsilon$ are arbitrary, this completes the proof of the upper bound. ∎

Remark 7.1*.*

It is clear that if the network satisfies Eq. 4.3 and $\zeta=0$ in Eq. 3.4, then the same conclusion for the lower bound can be drawn by invoking Proposition 6.2 in the preceding proof.

7.2. Proof of Theorem 3.2

Proof of the lower bound.

The proof follows by a similar argument as in the proof of the lower bound for Theorem 3.1. Let $\{Z^{n_{k}}\in{\mathfrak{Z}}^{n_{k}}\}\subset\bm{{\mathfrak{Z}}}$, with $\{n_{k}\}\subset\mathds{N}$ an increasing sequence, such that J_{\mathsf{o}}\bigl{(}\hat{X}^{n_{k}}(0),Z^{n_{k}}\bigr{)} converges to a finite value. Select an increasing sequence $\{T_{k}\}\subset\mathds{R}_{+}$ such that (7.5) holds with $J$ replaced by $J_{\circ}$ and $r$ by $r_{\circ}$. Following the proof of Theorem 3.1, let $\hat{\uppi}\in{\mathcal{P}}(\mathds{R}^{I}\times{\mathbb{U}})$ be the limit of $\Phi^{n_{k}^{\prime}}_{T_{k}^{\prime}}$ along some subsequence $\{n_{k}^{\prime},T_{k}^{\prime}\}\subset\{n_{k},T_{k}\}$. Recall the definition of $r_{j}$ in Eq. 3.5. Since $r_{j}$ is bounded below, taking limits, we obtain $\hat{\uppi}(r_{j})\;\leq\;\updelta_{j}$, $j\in{\mathcal{J}}$. Thus, by Lemmas 3.3–3.5 and Theorems 3.1–3.2 in [15], optimality implies that $\hat{\uppi}(r_{\mathsf{o}})\geq\varrho^{*}_{\mathsf{c}}$. As the proof of Theorem 3.1, we obtain,

[TABLE]

This proves the lower bound. ∎

Proof of the upper bound.

Let $\epsilon>0$ be given. By Corollary 7.1, there exists a continuous precise control $v_{\epsilon}\in\mathfrak{U}_{\mathrm{SSM}}$ and constants $\updelta^{\epsilon}_{j}<\updelta_{j}$, $j\in{\mathcal{J}}$, satisfying Eq. 7.1, and

[TABLE]

For the $n^{\rm th}$ system, we construct a Markov scheduling policy $Z^{n}$ as in the proof of the upper bound of Theorem 3.1, by concatenating $z^{n}[v_{\epsilon}]$ and $z\in{\mathscr{Z}}^{n}$ in Definition 5.1.

Following the proof of part (i) and choosing $\eta$ small enough, i.e., $\eta<\epsilon\wedge\frac{1}{2}\min\{\updelta_{j}-\updelta^{\epsilon}_{j},\;j\in{\mathcal{J}}\}$, we obtain

[TABLE]

This completes the proof of the upper bound. ∎

7.3. Proof of Theorem 4.2

Proof of the lower bound.

The proof follows along the same lines as that of Theorem 3.2, with the only difference that we use Proposition 6.2 instead of Proposition 6.1 to assert tightness of the ergodic occupation measures. With $\hat{\uppi}$ as given in that proof, we have

[TABLE]

We then obtain

[TABLE]

by Eq. 4.6 and the uniform integrability of

[TABLE]

which is asserted by Corollary 6.1. The proof is then completed using Eqs. 7.11 and 7.12 and Theorem 4.1 (e). ∎

Proof of the upper bound.

This also follows along the lines as that of Theorem 3.2. For the limiting diffusion control problem, by Theorem 5.7 and Remark 5.1 in [15], for any $\epsilon>0$, there exists a continuous precise control $v_{\epsilon}\in\mathfrak{U}_{\mathrm{SSM}}$ for (P3*′*) satisfying Eq. 7.1 and

[TABLE]

In addition, we have

[TABLE]

This follows from observing that $\{\uppi_{v_{\epsilon}}\,,\;\epsilon\in(0,1)\}$ is tight, and $(e\cdot x)^{-}$ is strictly positive on an open subset of $B_{1}$, and from applying the Harnack inequality for the density of the invariant probability measure of the limiting diffusion.

For the $n^{\rm th}$ system, we construct a Markov scheduling policy $\mathring{z}^{n}$ as in the proof of the upper bound of Theorem 3.2, and obtain

[TABLE]

The result then follows by Eqs. 7.13 and 7.14, thus completing the proof. ∎

8. Conclusion

In this work as well as in [14, 15, 16], we have studied ergodic control problems for multiclass multi-pool networks in the H–W regime under the hypothesis that at least one abandonment parameter is positive. The key technical contributions include (i) the development of a new framework of ergodic control (unconstrained and constrained) of a broad class of diffusions, (ii) the stabilization of the limiting diffusion and the diffusion-scaled state processes, and (iii) the technique to prove asymptotic optimality involving a spatial truncation and concatenation of scheduling policies that are stabilizing. The methodology and theory can be potentially used to study ergodic control of other classes of stochastic systems.

There are several open problems that remain to be solved. First, in this work, we have identified a class of BQBS stable networks as discussed in Section 4. It will be interesting to find some examples of network models in which the boundedness of the queueing process would not imply the boundedness of the state process. Second, we have studied the networks with at least one positive abandonment parameters. It remains to study the networks with no abandonment. The challenges lie in understanding the stability properties of both the limiting diffusions and the diffusion-scaled state processes. It is worth noting that the existence of a stabilizing control asserted in Theorem 2.1, which is established via the leaf elimination algorithm in [15], depends critically on the assumption that at least one abandonment parameter is positive. Although the proof of exponential ergodicity of the BSPs also relies on that assumption, this property is expected to hold with certain positive safety staffing for at least one server pool when all abandonment rates are zero.

Acknowledgements

This research was supported in part by the Army Research Office through grant W911NF-17-1-0019, and in part by the National Science Foundation through grants DMS-1715210 and DMS-1715875. In addition, the work of Ari Arapostathis was supported in part by the Office of Naval Research through grant N00014-14-1-0196.