E2E Delay Guarantee for the Tactile Internet via joint NFV and Radio Resource Allocation

TL;DR

This paper proposes a joint resource allocation scheme for the Tactile Internet that considers both radio and NFV resources to meet ultra-low end-to-end delay requirements, using advanced optimization techniques.

Contribution

It introduces the first joint R-RA and NFV-RA framework considering all delay components in a heterogeneous network for TI.

Findings

Significant reduction in network costs with joint optimization.

Effective delay guarantees for each user in TI.

Validation of proposed algorithms through simulations.

Abstract

The Tactile Internet (TI) is one of the next generation wireless network services with end to end (E2E) delay as low as 1~ms. Since this ultra low E2E delay cannot be met in the current 4G network architecture, it is necessary to investigate this service in the next generation wireless network by considering new technologies such as network function virtualization (NFV). On the other hand, given the importance of E2E delay in the TI service, it is crucial to consider the delay of all parts of the network, including the radio access part and the NFV core part. In this paper, for the first time, we investigate the joint radio resource allocation (R-RA) and NFV resource allocation (NFV-RA) in a heterogeneous network where queuing delays, transmission delays, and delays resulting from virtual network function (VNF) execution are jointly considered. For this setup, we formulate a new resource allocation (RA) problem to minimize the total cost function subject to guaranteeing E2E delay of each user. Since the proposed optimization problem is highly non-convex, we exploit alternative search method (ASM), successive convex approximation (SCA), and heuristic algorithms to solve it. Simulation results reveal that in the proposed scheme can significantly reduce the network costs compared to the case where the two problems are optimized separately.

Figures

Tables

Table 1. TABLE I : Computational Complexity

Subproblems	Computational Complexity
Subcarrier Allocation subproblem	$𝒪(z_1\times (U_{\text{Tot}}K+4U_{\text{Tot}}+U_{\text{Tot}}^2$ $+O_{\text{Tot}}L+4O_{\text{Tot}}+O_{\text{Tot}}^2))$
Power Allocation subproblem	$𝒪\left(\frac{\log \left(5U_{\text{Tot}}+J\right)/q^0\rho }{\log \zeta }\right)$
NFV Allocation subproblem	$𝒪\left(z_2\times \left(3U_{\text{Tot}}F+U_{\text{Tot}}^2{F}^2{J}^2+U_{\text{Tot}}\right)\right)$
Delay adjustment subproblem	$𝒪\left(\frac{\log \left(6U_{\text{Tot}}\right)/{\tilde{q}}^0\tilde{\rho }}{\log \tilde{\zeta }}\right)$

Table 2. TABLE II : Considered parameters in numerical results

Parameter	Value of each parameter
Coverage area	$10$$\sim$${\text{Km}}^2$
Path loss exponent	$3$
QoS exponent	$11$
PSD of the received AWGN noise	$-174$ dBm/Hz
Bandwidth of UL	$5$ MHz
Bandwidth of DL	$5$ MHz
Number of UL subcarriers	$8$
Number of DL subcarriers	$16$
Maximum transmit power of the MBS	$46$ dBm
Maximum transmit power of the SBS	$43$ dBm
Maximum transmit power of each tactile user	$23$ dBm
${\varrho }_1$	$1\$/\text{watts}$
${\varrho }_2$	$1\$/\text{ms}$

Equations

$$\begin{array}{rl}& x_{u_j^s}^k=\{\begin{array}{ll}1,& \text{If subcarrier k is assigned to tactile user ujs}\\ & \text{ at BS j with requested NS s},\\ 0,& \text{Otherwise}.\end{array}.\end{array}$$

$$\text{C1: }\sum _{s\in \mathcal{S}}\sum _{u_j^s\in {\mathcal{U}}_j^s}{x}_{u_j^s}^k\le 1,\forall j\in \mathcal{J},k\in \mathcal{K}.$$

$$\begin{array}{r}{r}_{u_j^s}=\sum _{k\in \mathcal{K}}{x}_{u_j^s}^k{\log }_2(1+{\gamma }_{u_j^s}^k),\forall u_j^s\in {\mathcal{U}}_j^s,s\in \mathcal{S},j\in \mathcal{J},\end{array}$$

$$I_{u_j^s}^k=\sum _{\genfrac{}{}{0ex}{}{m\in \mathcal{J},\text{hfill}}{m\ne j\text{hfill}}}\sum _{v\in \mathcal{S}}\sum _{u_m^v\in {\mathcal{U}}_m^v}{x}_{u_m^v}^k{p}_{u_m^v}^k{h}_{u_m^v,j}^k,$$

$$\forall u_j^s\in {\mathcal{U}}_j^s,s\in \mathcal{S},j\in \mathcal{J},k\in \mathcal{K}.$$

$$\sum _{j\in \mathcal{J}}\sum _{o_j\in {\mathcal{O}}_j}{\vartheta }_{o_j}^{u_m^s}\le 1,\forall u_m^s\in {\mathcal{U}}_m^s,m\in \mathcal{J},s\in \mathcal{S}.$$

$$\begin{array}{rl}& {\tau }_{o_j}^l=\\ & \{\begin{array}{ll}1,& \text{If subcarrier l is assigned to teleoperator oj at BS j},\\ 0,& \text{Otherwise}.\end{array}.\end{array}$$

$$\text{C2: }\sum _{o_j\in {\mathcal{O}}_j}{\tau }_{o_j}^l\le 1,\forall j\in \mathcal{J},l\in \mathcal{L}.$$

$$\begin{array}{rl}& {\hat{r}}_{o_j}=\sum _{l\in \mathcal{L}}{\tau }_{o_j}^l{\log }_2(1+{\hat{\gamma }}_{o_j}^l),\forall o_j\in {\mathcal{O}}_j,j\in \mathcal{J},\end{array}$$

$${\hat{I}}_{o_j}^l=\sum _{\genfrac{}{}{0ex}{}{m\in \mathcal{J},\text{hfill}}{m\ne j\text{hfill}}}\sum _{o_m\in {\mathcal{O}}_m}{\tau }_{t_m}^l{\hat{p}}_{t_m}^l{\hat{h}}_{o_j,m}^l,\forall o_j\in {\mathcal{O}}_j,j\in \mathcal{J},$$

$$\text{C3: }\sum _{k\in \mathcal{K}}{x}_{u_j^s}^k{p}_{u_j^s}^k\le P_{u_j^s}^{\text{Max}},\forall u_j^s\in {\mathcal{U}}_j^s,s\in \mathcal{S},j\in \mathcal{J}.$$

$$\text{C4: }\sum _{o_j\in {\mathcal{O}}_j}\sum _{l\in \mathcal{L}}{\tau }_{o_j}^l{\hat{p}}_{o_j}^l\le {\hat{P}}_j^{\text{Max}},\forall j\in \mathcal{J}.$$

$$\begin{array}{rl}& A_{u_j^s,f_s,n_2}^{u_m^w,f_w,n_1}\\ & =\{\begin{array}{ll}1,& \text{If NF }{f}_s\text{ of NS }s\text{ for user ujs at BS }{n}_2\text{ is executed}\\ & \text{ after NF }{f}_w\text{ of NS }w\text{ for user umw at BS }{n}_1,\\ 0,& \text{otherwise}.\end{array}.\end{array}$$

$$\text{C5: }\sum _{j^{\prime }\in \mathcal{J}}{A}_{u_j^s,f_s,j^{\prime }}^{u_m^w,f_w,n_1}\le 1,$$

$$\forall m,j\in \mathcal{J},u_m^w,u_j^s\in {\mathcal{U}}_j^s,s,w\in \mathcal{S},f_s\in {\mathcal{F}}_s,f_w\in {\mathcal{F}}_w.$$

$$\text{C6: }{D}_{u_j^s}^{\text{T,UL}}+D_{u_j^s}^{\text{T,DL}}+D_{u_j^s}^{\text{NFs}}+D_{u_j^s}^{\text{Q,UL}}+D_{u_j^s}^{\text{Q,DL}}\le D_{u_j^s}^{\max },$$

$$\forall u_j^s\in {\mathcal{U}}_j^s,j\in \mathcal{J},s\in \mathcal{S},$$

$$\text{C7: }{D}_{u_j^s}^{\text{T,UL}}\ge \frac{C_{u_j^s}}{r_{u_j^s}},u_j^s\in {\mathcal{U}}_j^s,s\in \mathcal{S},j\in \mathcal{J},$$

$$\text{C8: }{D}_{u_j^s}^{\text{T,DL}}\ge \frac{C_{u_j^s}}{{\sum }_{m\in \mathcal{J}}{\sum }_{o_m\in {\mathcal{O}}_m}{\vartheta }_{t_m}^{u_j^s}{\hat{r}}_{t_m}},$$

$$\forall u_j^s\in {\mathcal{U}}_j^s,s\in \mathcal{S},j\in \mathcal{J},$$

$$\text{C9: }{A}_{u_j^s,f_s,n_2}^{u_m^b,f_b,n_1}{\omega }_{u_j^s}^f\ge A_{u_j^s,f_s,n_2}^{u_m^b,f_b,n_1}({\omega }_{u_m^b}^{f_b}+{\beta }_{n_2}^{f_s}\frac{C_{u_j^s}}{{\Omega }_{n_2}}+\text{mathds}{1}_{n_2\ne n_1}\frac{C_{u_j^s}}{{\Psi }_{n_1}^{n_2}}),$$

$$\forall u_j^s\in {\mathcal{U}}_j^s,u_m^b\in {\mathcal{U}}_m^b,b,s\in \mathcal{S},f_s\in {\mathcal{F}}_s,f_b\in {\mathcal{F}}_b,$$

$$n_1,n_2,j,m\in \mathcal{J},$$

$$\text{C10: }{D}_{u_j^s}^{\text{NFs}}\ge {\omega }_{u_j^s}^{F_s},\forall u_j^s\in {\mathcal{U}}_j^s.$$

$$B_{u_j^s}={\Lambda }_{u_j^s}\frac{(e^{{\theta }_{u_j^s}}-1)}{{\theta }_{u_j^s}},{\forall }_{u_j^s}\in {\mathcal{U}}_j^s,s\in \mathcal{S},j\in \mathcal{J},$$

$${ϵ}_{u_j^s}^{\text{UL}}=\mathrm{Pr}\{D_{u_j^s}^{\text{UL}}>D_{u_j^s}^{\text{Q,UL}}\}={\eta }_1\exp (-{\theta }_{u_j^s}{B}_{u_j^s}{D}_{u_j^s}^{\text{Q,UL}})\le {\delta }_1,$$

$$\forall u_j^s\in {\mathcal{U}}_j^s,s\in \mathcal{S},j\in \mathcal{J},$$

$$\begin{array}{rl}& \exp (-{\theta }_{u_j^s}{B}^j{D}_{u_j^s}^{\text{Q,UL}})=\exp (-{\theta }_{u_j^s}{\Lambda }_{u_j^s}\frac{(e^{{\theta }_{u_j^s}}-1)}{{\theta }_{u_j^s}}{D}_{u_j^s}^{\text{Q,UL}})=\\ & \exp (-{\Lambda }_{u_j^s}(e^{{\theta }_{u_j^s}}-1)D_{u_j^s}^{\text{Q,UL}})\le {\delta }_1.\end{array}$$

$$\begin{array}{r}\text{C11: }{r}_{u_j^s}\ge \frac{\ln (1/{\delta }_1)}{(e^{{\theta }_{u_j^s}}-1)D_{u_j^s}^{\text{Q,UL}}},{\forall }_{u_j^s}\in {\mathcal{U}}_j^s,s\in \mathcal{S},j\in \mathcal{J}.\end{array}$$

$${ϵ}_{u_j^s}^{\text{DL}}=\mathrm{Pr}\{D_{u_j^s}^{\text{DL}}>D_{u_j^s}^{\text{Q,DL}}\}={\eta }_2\exp (-{\tilde{\theta }}_{u_j^s}{\tilde{B}}_{u_j^s}{D}_{u_j^s}^{\text{Q,DL}}),$$

$${\forall }_{u_j^s}\in {\mathcal{U}}_j^s,s\in \mathcal{S},j\in \mathcal{J},$$

$$\begin{array}{rl}& \exp (-{\tilde{\theta }}_{u_j^s}{\tilde{B}}_{u_j^s}{D}_{u_j^s}^{\text{Q,DL}})=\exp (-{\tilde{\theta }}_{u_j^s}{\tilde{\Lambda }}_{u_j^s}\frac{(e^{{\tilde{\theta }}_{u_j^s}}-1)}{{\tilde{\theta }}_{u_j^s}}{D}_{u_j^s}^{\text{Q,DL}})=\\ & \exp (-{\tilde{\Lambda }}_{u_j^s}(e^{{\tilde{\theta }}_{u_j^s}}-1)D_{u_j^s}^{\text{Q,DL}})\le {\delta }_2.\end{array}$$

$$\sum _{m\in \mathcal{J}}\sum _{o_m\in {\mathcal{O}}_m}{\vartheta }_{o_m}^{u_j^s}{\hat{r}}_{o_m}\ge \frac{\ln (1/{\delta }_2)}{(e^{{\tilde{\theta }}_{u_j^s}}-1)D_{u_j^s}^{\text{Q,DL}}},$$

$$\forall u_j^s\in {\mathcal{U}}_j^s,s\in \mathcal{S},j\in \mathcal{J}.$$

$$\displaystyle C=\varrho_{1}\sum_{j\in\mathcal{J}}\bigg{(}\sum_{s\in\mathcal{S}}\sum_{{u^{s}_{j}}\in\mathcal{U}^{s}_{j}}\sum_{k\in\mathcal{K}}x_{u^{s}_{j}}^{k}p_{u^{s}_{j}}^{k}+\sum_{o_{j}\in\mathcal{O}_{j}}\sum_{l\in\mathcal{L}}\tau_{o_{j}}^{l}\hat{p}_{o_{j}}^{l}\bigg{)}+$$

Full text

I Introduction

I-A State of the Art

In the next generation wireless networks, heterogeneous services with different requirements are proposed. These services are generally classified as enhanced mobile broadband (eMBB), massive machine-type communications (mMTC), and ultra-reliable and low latency communications (URLLC). eMBB requires high data rate while mMTC requires high number of connections and the data rate of each connection is low. URLLC such as the Tactile Internet (TI), requires ultra low end-to-end (E2E) delay and ultra high reliability [1, 2].

The TI requires an E2E delay of about $1$ ms to transmit the sense of touch remotely. The TI has many applications, including remote monitoring, remote surgery, distance education, and remote driving. Due to the importance of E2E delay in the TI service, it is necessary to consider all delays in the network to satisfy this requirement [3, 4, 5, 6].

Based on current practice in the networks, each of the services requires special physical equipment and servers at base stations (BSs). Therefore, to provide a new service, it is necessary to deploy the required physical equipment (middle-boxes) such as special servers, which are not cost-effective for the operators. Moreover, providing space and energy for various middle-boxes and managing them are costly. Network function virtualization (NFV) is a promising method to deal with these bottlenecks [7, 8, 9].

NFV reduces capital expenditure (CAPEX) and operational expenditure (OPEX) and increases the management capabilities while optimizing the utilization of network equipment and servers, e.g., middle-boxes. The purpose of NFV is to shift middle-box processing from hardware to software [10]. In NFV, network functions (NF) are implemented virtually on the servers which are called virtual network functions (VNFs). NFV facilitates the installation and implementation of VNFs on the various servers at any place and time in the network [11].

In order to deploy a network service (NS), it is necessary to pass the traffic through a set of middle-boxes (i.e. VNFs) in a specific order, each performing a particular operation. Choosing suitable middle-boxes and steering traffic between them are two important key challenges in NFV-based systems. Hence, for each NS, the number of VNFs, their chains, and the location of each VNF on the servers should be determined [10, 11]. Thus, one of the challenges in NFV deployment is how to combine, locate, and schedule VNFs to realize an NS.

The NFV resource allocation (NFV-RA) includes three stages: 1) VNF composition to execute an NS that is called chain composition, 2) virtual links and servers allocation which is called VNF embedding, and 3) VNF scheduling to execute an NS [12, 13]. In many cases, the allocation of resources is done such that the execution time of NF’s is minimized. For delay-sensitive services, NFV-RA should be performed such that the amount of delay is kept less than a threshold value. Therefore, NFV-RA is very important for delay-sensitive services and it is necessary to consider all three stages jointly.

In [14, 15, 16], the VNF placement optimization is studied by considering the maximum link utilization. VNF placement and scheduling are investigated jointly in [17] by considering the buffer capacity of nodes and the processing time of VNFs. The authors in [18] study joint VNF composition and placement to minimize bandwidth utilization by keeping the overall runtime below the threshold value. Joint optimization of three stages of NFV-RA is studied in [19] by considering network cost and service performance.

In wireless cellular networks, allocation of the radio resources such as power, bandwidth, and resource blocks should be optimized according to the service requirements needed for multiple users (e.g., data rate and delay). In practice, Radio Resource Allocation (R-RA) plays a key role for guaranteeing the quality of service (QoS). Given that the TI is highly sensitive to delay, R-RA becomes even more important in this case [3, 4]. In [3], a cross-layer optimization is proposed for guaranteeing the QoS parameters such as queuing delay and reliability of the TI service. In that work, the QoS requirement is satisfied by considering the transmit power and bandwidth constraints. In [20], both queuing delay and uplink (UL)/downlink (DL) transmission delays are considered in a system with single roadside BS. In this work, energy efficiency (EE) is maximized by considering the QoS requirements and optimal resource allocation. UL transmission design with massive machine type devices in the TI is studied in [21]. In that work, the authors investigate the effect of diversity on reliability and the QoS for the TI service. A R-RA scheme for the TI in LTE-A network is investigated in [22] for the first time in which single carrier frequency division multiple access (SC-FDMA) and orthogonal frequency division multiple access (OFDMA) are considered for UL and DL transmission, respectively. In [23], a R-RA scheme for the TI in the single cell network based on sparse code multiple access (SCMA) is proposed in which queuing delay both at the source and the base station is considered. In [24, 25], a R-RA for the TI in the cloud radio access network (C-RAN) architecture of the next generation wireless network is proposed. In that work, a multi cell network assisted via power domain non-orthogonal multiple access (PD-NOMA) is considered.

I-B Our Contributions

Since the TI is supposed to be implemented in the next generation of wireless networks, it is of great importance that it is studied and analyzed in the framework of the next generation of wireless network technologies. On the other hand, the TI is a delay-sensitive service, and hence all of the parameters that affect the E2E delay should be considered. This includes queuing delays, transmission delays, and delays resulting from the execution of NFs. This means that current frameworks in which R-RA and NFV-RA are considered separately, may not provide an efficient allocation of the network resources. In other words in this case, more resources might be necessary to meet the E2E delay requirements. As such, joint R-RA and NFA-RA has to be implemented.

In this paper, we consider a joint R-RA and NFV-RA framework in a heterogeneous cellular network. In this system model, we consider multiple users with different types of services in which our target is to minimize the total cost subject to E2E delay constraints. The contributions of this paper can be summarized as follows:

• •

We consider and analyze the requirements of TI Service in an NFV-based framework for the first time.

• •

We perform a novel joint R-RA and NFV-RA for the TI service, referred to as the joint approach (JA).

• •

Using the proposed framework, we show a considerable amount of costs in term of transmit power and NFV execution time saving compared to the separate approach (SA).

• •

We consider a practical scenario in which each user can request a service with its own delay requirement. In prior works, heterogeneity of services are not considered.

• •

In this paper, we propose a general solution for placement and scheduling of VNFs by considering delay requirements and radio resource limitations.

I-C Paper Organization

The rest of this paper is as follows. In Section II, the system model and related constraints are described. In Section III, we propose the joint optimization problem and its solution. In Section IV, computational complexity is calculated. Numerical results are presented in Section V. Finally, Section VI concludes the paper.

II System Model

II-A General Description

We consider a heterogeneous cellular network where all BSs are connected together via backhaul links. Furthermore, there exist multiple tactile users and teleoperators where each tactile user sends its data to its paired teleoperator111The teleoperator is a tactile data receiver such as actuator that executes a command based on received data., via UL and DL transmission links. As shown in Fig. 1, we consider a heterogeneous network consisting of a macro BS (MBS) and several small BSs (SBS) denoted by $\mathcal{J}^{\prime}=\{1,...,J\}$. We indicate the MBS with index [math]. Therefore, $\mathcal{J}=\{0,1,...,J\}$ is the set of all BSs. Moreover, in the proposed system model, there is a set of $\boldsymbol{\mathcal{{O}}_{j}}=\{1,\dots,O_{j}\}$ teleoperators at BS $j$ which receive the tactile data. The total number of teleoperators in our system model is equal to $\mathcal{O}_{\text{Tot}}=\bigcup_{j\in\mathcal{J}}\mathcal{O}_{j}$. In addition, in our system model, the set of all subcarriers for UL transmission and DL transmission are denoted by $\boldsymbol{\mathcal{K}}=\{1,\dots,K\}$ and $\boldsymbol{\mathcal{L}}=\{1,\dots,L\}$, respectively.

It should be noted that the proposed system model works in frequency division duplexing (FDD) mode. In this system model, we consider orthogonal frequency division multiple access (OFDMA) in which each subcarrier is allocated to at most one user in each time slot.

Given that NFs of one NS are located in different BSs, for executing each NS, the data should be transferred between BSs. We assume that there exist $\mathcal{S}\in\{1,...,S\}$ NSs in our considered system model. NS $s$ consists of a set of NFs $\mathcal{F}_{s}=\{1,...,F_{s}\}$. We assume that the chain of NFs of each NS is known and fixed and we focus on scheduling and placement of NFs. In the proposed system model, $\mathcal{U}^{s}_{j}=\{1,...,U^{s}_{j}\}$ tactile users request service $s$ at BS $j$ and the total number of tactile users is equal to $\mathcal{U}_{\text{Tot}}=\bigcup_{j\in\mathcal{J}}\bigcup_{s\in\mathcal{S}}\mathcal{U}^{s}_{j}$. The proposed joint R-RA and NFV-RA is an offline algorithm i.e., at the beginning of each frame after all users send their requests, joint R-RA and NFV-RA should be performed for all received requests. It is worth noting that we assume the channel power gain is constant in each frame and varies frame by frame.

Remark 1**.**

In this paper, we assume that the Backhaul topology is full mesh [26, 27, 28, 29, 30] and we only consider the capacity of the backhaul links regardless of its type (Fiber/mmWave).

Remark 2**.**

It is assumed that the next generation wireless radio access network (RAN) is virtual enabled meaning that some RAN functions are implemented virtually including Virtualized BS [31, 32]. Given that the focus of this paper is on the delay-sensitive TI service, it is necessary that some processing functions, such as mobile edge computing (MEC) functions, are executed at each BS [33, 34]. Therefore, we consider a lightweight server at each SBS and a high volume server (HVS) at the MBS [35, 33, 34].

II-B * Radio Resource Allocation*

We consider a binary variable $x_{u^{s}_{j}}^{k}$ which is set to 1 if subcarrier $k$ is assigned to tactile user $u^{s}_{j}$ at BS $j$ with requested service $s$ and otherwise it is set to 0 , i.e.,

[TABLE]

Given that we use OFDMA in this setup, we have the following constraint in which each subcarrier is assigned to at most one user:

[TABLE]

The achievable rate of tactile user $u^{s}_{j}$ on subcarrier $k$ at BS $j$ can be calculated as

[TABLE]

where $\gamma^{k}_{u^{s}_{j}}=\frac{p_{u^{s}_{j}}^{k}h_{u^{s}_{j}}^{k}}{\sigma_{u^{s}_{j}}^{k}+I_{u^{s}_{j}}^{k}}$ is signal to interference plus noise ratio (SINR), in which $p_{u^{s}_{j}}^{k}$, $h_{u^{s}_{j}}^{k}$, and $\sigma_{u^{s}_{j}}^{k}$ represent the transmit power, channel power gain, and noise power from BS $j$ to user $u^{s}_{j}$ on subcarrier $k$ for service $s$, respectively. $I_{u^{s}_{j}}^{k}$ is inter-cell interference which is equal to

[TABLE]

where $h_{u_{m}^{v},j}^{k}$ is the channel power gain between BS $j$ and user $u_{m}^{v}$ over subcarrier $k$. We introduce $\vartheta_{o_{j}}^{u^{s}_{m}}\in\{0,1\}$ which is set to $1$ if teleoperator $o_{j}$ at BS $j$ is paired with tactile user $u^{s}_{m}$ at BS $m$ with requested service $s$ i.e., tactile user $u^{s}_{m}$ sends its data via UL transmission link to teleoperator $o_{j}$ over DL transmission link, which is paired. Since we adopt FDD, the sets of UL and DL subcarriers are different. As each tactile user has at most one paired teleoperator, we have

[TABLE]

For teleoperators, we consider a binary variable $\tau_{o_{j}}^{l}$ which is set to 1 if subcarrier $l$ is assigned to teleoperator $o_{j}$ at BS $j$ and otherwise is set to 0, i.e.,

[TABLE]

Again, due to using OFDMA, we have the following constraint:

[TABLE]

The achievable rate of teleoperator $o_{j}$ at BS $j$ can be calculated as

[TABLE]

where $\hat{\gamma}^{l}_{o_{j}}=\frac{\hat{p}_{o_{j}}^{l}\hat{h}_{o_{j}}^{l}}{\hat{\sigma}_{o_{j}}^{l}+\hat{I}_{o_{j}}^{l}}$ is SINR of teleoperator $o_{j}$ on subcarrier $l$, in which $\hat{p}_{o_{j}}^{l}$, $\hat{h}_{o_{j}}^{l}$, and $\hat{\sigma}_{o_{j}}^{l}$ represent the transmit power, channel power gain, and noise power from teleoperator $o_{j}$ to BS $j$ over subcarrier $l$, respectively. $\hat{I}_{o_{j}}^{l}$ is inter-cell interference which is equal to

[TABLE]

where $\hat{h}_{{o_{j}},m}^{l}$ is the channel power gain between teleoperator $o_{j}$ and BS $j$.

Due to the power limitation of each tactile user in UL transmission, we have

[TABLE]

where $P_{u^{s}_{j}}^{\text{Max}}$ is the maximum transmit power of tactile user ${u^{s}_{j}}$. Moreover, due to the power limitation of each BS, we have

[TABLE]

where $\hat{P}_{j}^{\text{Max}}$ is the maximum transmit power of BS $j$.

II-C NFV Resource Allocation

In this paper, we propose a general solution for placement and scheduling of VNFs. An example of VNF placement and scheduling is shown in Fig. 2. To achieve this goal, we introduce a binary variable $A_{u^{s}_{j},f_{s},n_{2}}^{u^{w}_{m},f_{w},n_{1}}\in\{0,1\}$ which is set to $1$ if NF $f_{s}$ of NS $s$ for tactile user $u^{s}_{j}$ is executed at BS $n_{2}$ after NF $f_{w}$ of NS $w$ for tactile user $u^{w}_{m}$ was executed at BS $n_{1}$ and is set to [math] otherwise, i.e.,

[TABLE]

It is assumed that each NF of each NS is performed at only one BS. Therefore, we have:

[TABLE]

II-D Delay Model

The total delay for tactile user ${u^{s}_{j}}$ in our system model consists of five components as shown in Fig. 3, namely, 1) UL transmission delay in access ($D_{u^{s}_{j}}^{\text{T,UL}}$), 2) UL queuing delay ($D_{u^{s}_{j}}^{\text{Q,UL}}$), 3) delay resulting from executing NFs ($D_{u^{s}_{j}}^{\text{NFs}}$), 4) DL transmission delay in access ($D_{u^{s}_{j}}^{\text{T,DL}}$), and 5) DL queuing delay ($D_{u^{s}_{j}}^{\text{Q,DL}}$). Due to the delay constraint for each NS, we have

[TABLE]

where $D_{u^{s}_{j}}^{{\rm{max}}}$ is maximum allowable E2E delay.

II-D1 Transmission Delay

To calculate the UL and DL transmission delays, we have

[TABLE]

[TABLE]

where $C_{u}^{s}$ is the total transmitted bits of user $u^{s}_{j}$.

II-D2 Delays of NS

To calculate the total delay of each NS for each user, we introduce a continuous variable $\omega_{u^{s}_{j}}^{f}$ which is set to end of the execution time of NF $f$ at BS $j$ for user $u^{s}_{j}$ with NS $s$. In addition, we assume the link between BSs has limited capacity $\Psi_{n_{1}}^{n_{2}},\forall n_{1},n_{2}\in\mathcal{J}$. Therefore, we have the following constraint

[TABLE]

where $\beta_{n_{2}}^{f_{s}}$ is the data processing coefficient of NF $f_{s}$ in BS $n_{2}$ and $\varOmega_{j}$ is the processing rate of BS $j$ (bit/s). In this paper, we assume that the service requests of all users are received, then the scheduling and placement of VNFs are performed. Therefore, the delays associated to execution of NFs for each user is equal to the time when the last function is executed. The total delay resulting from executing each NS is calculated as

[TABLE]

II-D3 Queuing Delay

The queuing delay consists of two components: 1) UL queuing delay and 2) DL queuing delay, as shown in Fig. 3. Based on [3], the aggregation of arrival bit rate of users can be modeled as a Poisson process. For a Poisson arrival process, the effective bandwidth of tactile user $u^{s}_{j}$ is obtained as [36, 3]

[TABLE]

where $\theta_{u^{s}_{j}}$ is the statistical QoS exponent of tactile user ${u^{s}_{j}}$. A larger $\theta_{u^{s}_{j}}$ indicates a more stringent QoS and a smaller $\theta_{j}$ implies a looser QoS requirement. $\Lambda_{u^{s}_{j}}$ is the number of bits arrived in time unit at BS $j$ queue from tactile user ${u^{s}_{j}}$ defined as ${\Lambda_{u^{s}_{j}}}=r_{u^{s}_{j}},~{}\forall{u^{s}_{j}}\in\mathcal{U}^{s}_{j}$. In the TI service, queuing delay violation probability should be ultra-low [3]. Therefore, queuing delay violation probability for tactile user $u^{s}_{j}$ in UL transmission is obtained as follows

[TABLE]

where $D_{u^{s}_{j}}^{\text{UL}}$ is the ${u^{s}_{j}}^{\text{th}}$ tactile user queuing delay in UL, $D_{u^{s}_{j}}^{\text{Q,UL}}$ is the maximum queuing delay in UL, $\eta_{1}$ is the non-empty buffer probability in UL, and $\delta_{1}$ is the maximum queuing delay violation probability in UL. Equation (II-D3) can be simplified to

[TABLE]

Therefore, we have222In this work, we have two buffers for each user: the first one is for the uplink (UL) transmission and the second one is for the downlink (DL) transmission. For the queue in the UL transmission, we model the delay based on the effective bandwidth which leads to a constraint on the UL rate and for the second one we model the delay based on the effective capacity which leads to a constraint on DL rate. This model is similar to [22, 37, 38].

[TABLE]

Similarly, in DL transmission, we have

[TABLE]

where $D_{u^{s}_{j}}^{\text{DL}}$ is the ${u^{s}_{j}}^{\text{th}}$ tactile user at BS $j$ queuing delay in DL, $D_{u^{s}_{j}}^{\text{Q,DL}}$ is the maximum queuing delay in DL, $\eta_{2}$ is the non-empty buffer probability in DL, and $\delta_{2}$ is the maximum queuing delay violation probability in DL. Therefore, Equation (II-D3) can be simplified to

[TABLE]

Therefore, we have

[TABLE]

III Optimization Problem Formulation and Solution

In the proposed system model, we define the cost function $C$ as the total amount of physical and virtual network resource occupied for resource allocation, i.e.,

[TABLE]

where $\varrho_{1}$ and $\varrho_{2}$ are constant values for scaling the transmit power and execution time of NFs and have units such that the overall costs can be expressed in terms of monetary value. In the simulation setup, we set $\varrho_{1}$ to 1$/watts and$\varrho_{2}$to 1$/milliseconds. It is worth noting that the exact choices of the weights $\varrho_{1}$ and $\varrho_{2}$ depend on the business model of the network operator. The aim is to minimize the cost function by considering an E2E delay constraint and limitation of radio resources in the network. Based on the mentioned constraints C1-C12, the optimization problem can be written as

[TABLE]

The optimization variables in (III) are subcarrier allocation, power allocation, NFV allocation, time allocation, and delay adjustment for different users in access as well as in both UL and DL where $\boldsymbol{P}$, $\boldsymbol{X}$,$\boldsymbol{T}$, $\boldsymbol{W}$, $\boldsymbol{A}$, and $\boldsymbol{D}$ are the transmit power, the tactile users subcarrier allocation, teleoperators subcarrier allocation, end time vector of VNF executions, VNF scheduling vector, and delay adjustment, respectively. In problem (III), the rate function is a non-convex function, which leads to the non-convexity of the problem. In addition, this problem contains both discrete and continuous variables, which makes the problem more challenging.

The optimization problem (III) is mixed-integer non-convex programming and hard to solve. In order to facilitate the solution of this problem, we define new variables as $\dot{p}_{u^{s}_{j}}^{k}=x_{u^{s}_{j}}^{k}p_{u^{s}_{j}}^{k}$ and $\ddot{p}_{o_{j}}^{l}=\tau_{o_{j}}^{l}\hat{p}_{o_{j}}^{l}$. Therefore, (III) is turned into the following problem

[TABLE]

To solve problem (III), we exploit alternative search method (ASM) which is described in Algorithm 1 [39, 40]. Thus, the approach is to divide the problem into 4 subproblems: 1) Subcarrier allocation subproblem, 2) Power allocation subproblem, 3) NFV-RA subproblem, 4) Delay adjustment subproblem. These subproblems are solved iteratively, in which $z$ represents the iteration number. Please note that although the optimization process is performed thorough solving 4 subproblems, it is performed sequentially within an iterative framework such that the solution of each subproblem directly affects that of the next. As such, the resources are still allocated jointly in contrast to the separate approach (SA) case.

First, the initial values for power allocation, subcarrier allocations, NFV-RA, and delay adjustment should be obtained. Next, in each iteration, we obtain the optimal solution for the subcarrier allocation subproblem with $\dot{p}^{(z)}=\dot{p}^{0}$ and $\ddot{p}^{(z)}=\ddot{p}^{0}$. Then an optimal solution for the power allocation subproblem is obtained with $X^{(z)}=x^{0}$ and $T^{(z)}=\tau^{0}$ and the solution is fed as the initial value tonb the next subproblem. Later in the NFV-RA subproblem, we obtain a solution of the VNF placement and scheduling to satisfy problem (III) constraints. After that, we solve the delay adjustment subproblem and delay is adjusted by solving an integer linear programming (ILP) problem. The iteration stops when the iteration number exceeds a predetermined value $Z_{\text{TH}}$ or $||\boldsymbol{\dot{P}}^{(z)}-\boldsymbol{\dot{P}}^{(z-1)}||\leq\epsilon_{\text{TH}}$ and $||\boldsymbol{\ddot{P}}^{(z)}-\boldsymbol{\ddot{P}}^{(z-1)}||\leq\epsilon_{\text{TH}}$, where $||.||$ represents the vector Euclidean norm. The output of the last iteration is the suboptimal solution of Problem (III). It should be noted that there is no local or global optimality guarantee for ASM.

III-A Subcarrier Allocation Subproblem

The subcarrier allocation subproblem is as follows:

[TABLE]

In this subproblem, if the subcarrier with the highest channel gain is allocated to each user, the total transmit power is reduced and the total rate is increased which affects the delay (by increasing the total rate, the delay is reduced). Therefore, to avoid this issue, we propose a heuristic approach summarized in Algorithm 2.

III-B Power Allocation Subproblem

The power allocation subproblem is as follows:

[TABLE]

In (11), the constraints containing the rate function are non-convex. We convert them to convex constraints by applying the difference of two concave functions (DC) approximation which is based on the successive convex approximation (SCA) approach [41]. Hence, we write the rate function as follows:

[TABLE]

where

[TABLE]

[TABLE]

For each user $u^{s}_{j}$, we employ the following linear approximation based on first order Taylor series in the point ${\boldsymbol{\dot{P}}}^{z}$ as follows

[TABLE]

where $z$ indicates the iteration numbers and $\nabla g(\boldsymbol{\dot{P}}^{z-1})$ is obtained as follows:

[TABLE]

III-C NFV Resource Allocation Subproblem

The NFV-RA subproblem is as follows:

[TABLE]

Similar to the subcarrier allocation subproblem, we use a heuristic method described in Algorithm 3. It is worth noting that in JA approach, the decision (change the order) is made based on constraint C6 and in SA approach, the decision is made based on constraint C10.

III-D Delay Adjustment Subproblem

The delay adjustment subproblem is as follows:

[TABLE]

The delay adjustment subproblem is a LP problem and can be solved by any existing optimization toolbox such as CVX in MATLAB.

Here, we provide a discussion on the convergence of the proposed solution. For the solution to converge, it is necessary to examine the convergence of Algorithm 1 and the convergence of each subproblem solution. In Algorithm 1, at first, the subcarrier subproblem is solved. In this subproblem, to each user the subcarrier with maximum channel gain is allocated according to the initial power value. The ordering of subcarrier allocation is based on the priority of the requested service by the user in terms of the E2E delay requirement. If the delay constraints are not met for some users, the priority of those users will change. It should be noted that this subproblem does not directly affect the objective function as far as convergence of main problem is concerned. Next, the power allocation subproblem is solved using the SCA method which is a convergent algorithm [40, 42]. Therefore, in each iteration, we have $f({\bf{\dot{P}}}^{z-1},{\bf{\ddot{P}}}^{z-1})\geq f({\bf{\dot{P}}}^{z},{\bf{\ddot{P}}}^{z})$. The next subproblem is the NFV-RA subproblem. The VNFs placement and scheduling are performed based on NS priority in terms of E2E delay. It should be noted that NFV-RA subproblem does not directly affect the objective function as far as convergence of the main problem is concerned. The last subproblem is the delay adjustment subproblem whose convergence is intuitive. Therefore, in each iteration, a feasible value for power is obtained, which is less than or equal to the feasible value in the previous iteration and the overall algorithm converges in the achieved performance (not necessarily in the optimization variables) [43, 40, 42].

IV Computational complexity

In this section, the computational complexity of the proposed RA algorithm (Algorithm. 1) is calculated. The overall RA algorithm consists of four stages: (1) the subcarrier allocation subproblem (10), (2) the power allocation subproblem (11), (3) the NFV-RA subproblem (LABEL:NFVP-1), and (4) the delay adjustment subproblem which are jointly considered through an iterative approach. The computational complexity of each subproblem is shown in Table I where $z_{1}$ and $z_{2}$ are the number of iterations in corresponding subproblem [42, 44]. For power allocation subproblem and delay adjustment subproblem which are solved based on interior point method (IPM) by CVX in MATLAB, the computational complexity can be obtained as follows:

[TABLE]

where $Q_{\text{Constr}}$ is equal to the total number of constraints, $0\leq\rho\leq 1$ is the stopping criterion of IPM, $\zeta$ is used to update accuracy of the IPM, and $q^{0}$ is initial point for approximating the accuracy of IPM.

V Simulation Results

V-A Simulation Setup

In this section, the simulation results are presented to evaluate the performance of the proposed system model. In our simulations, the SBSs are located at an equal distance from MBS. The coverage area is considered $10$ $\text{Km}^{2}$. Moreover, we consider a Rayleigh fading wireless channel in which the channel power gain of subcarriers are independent. Channel power gains for the radio access links are set as $h_{u^{s}_{j}}^{k}=\Omega_{u^{s}_{j}}^{k}{d_{u^{s}_{j}}}^{\alpha}$ where $d_{u^{s}_{j}}$ is the distance between user $u^{s}_{j}$ and BS $j$, $\Omega_{u^{s}_{j}}^{k}$ is a random variable with Rayleigh distribution, and $\alpha=3$ is the path-loss exponent. The power spectral density (PSD) of the additive white Gaussian noise (AWGN) is set to be $-174$ dBm/Hz. The system model parameters are summarized in Table II [45].

V-B Performance Analysis

At first, we study the effects of network parameters on the performance of the proposed system model. In Fig. 4, the total cost is shown versus the number of tactile users in each BS for different NS requirements. As can be seen, by increasing the number of tactile users, the total cost increases. Moreover, by increasing the number of BSs, the number of served tactile users and subsequently the total cost increases. The most important point is that as the NS E2E delay is decreased, the amount of total transmit power is increased and consequently the total cost is increased. Now if all tactile users request only one type of service with specific E2E delay requirement (e.g., $10$ ms or $1$ ms), for services with $1$ ms E2E delay requirement in the fixed-point, e.g., $J=4$ and $U=5$, the total cost is about $9.3$ while in the setup with $5$ ms E2E delay requirement, the total cost is about $2.7$. Therefore, by decreasing the E2E delay requirement, the total transmit power increases drastically, and hence the total cost increases.

Fig. 5 illustrates the total cost versus the number of subcarriers. As can be seen, the total cost decreases by increasing the number of subcarriers. This is due to the fact that increasing the number of subcarriers, increases the diversity gain of the network which depends on channel model and frequency selectivity, i.e. the power delay profile. Therefore, the network has a higher degree of freedom in the subcarrier assignment and for each user, it can choose better subcarriers and in the same condition, less transmit power is needed. As a result, the total cost is reduced.

In Fig. 6, the effect of E2E delay of services on the total cost is addressed. In this regard, we assume all users request only one type of service with specific E2E delay requirement. Based on C11 and C12, by increasing the delay, the required rate is decreased, hence the total transmit power is reduced. Therefore, as can be seen from Fig. 6, the total cost decreases by increasing E2E delay of services.

V-C Comparison between Joint Approach (JA) and Separate Approach (SA)

In order to compare the proposed approach, i.e., JA, with the case where NFV-RA and R-RA are treated independently, i.e., SA, we assume a setting in which we decompose NFV-RA and R-RA and solve them separately. In R-RA, we set NFV delay manually in constraints C6 to fixed amount, e.g, $D_{u^{s}_{j}}^{\text{NFs}}=0.5$ ms. The new problem associated to R-RA in the SA can be written as

[TABLE]

This new optimization problem can be solved with the ASM and SCA methods as well as the CVX toolbox. Similarly, the new problem for NFV-RA is as follows:

[TABLE]

The NFV-RA problem can be solved with a heuristic Algorithm 3.Now we set the E2E delay requirements to 5 ms and 1 ms and compare the cost associated to SA and JA. For both cases in SA, the delay associated to the NFV problem is set to 0.5 ms. As can be seen, there is not much difference between the coast associated to JA and SA for E2E delay of 5 ms. The reason is that the constraint for the R-RA case is only slightly modified compared to the JA. For E2E delay of 1 ms, however, we observe cost increase as high as 50% when we choose SA over JA and the difference grows linearly when the number of tactile users.333This result is obtained based on the assumption that the ratio of $\varrho_{2}$ to $\varrho_{1}$ is 1. The realistic values for these 2 parameters depend on the actual network implementation. However, our simulation shows that the advantage of JA over SA remains significant for other values of $\varrho_{2}/\varrho_{1}$.

VI Conclusion

In this paper, we considered a heterogeneous wireless network with a variety of NSs for the Tactile Internet where we considered queuing delays, transmission delays, and delays resulting from VNF execution We then considered the joint radio resource allocation and NFV resource allocation to minimize the total cost function subject to guaranteeing E2E delay of each user. To solve this non-convex proposed resource allocation problem, we applied an ASM algorithm with SCA method. Simulation results reveal that for smaller values of the E2E delay (e.g. 1 ms), the proposed joint scheme can significantly reduce the network costs compared to the case where the two problems are optimized separately.