Performance Evaluation of Optimal Radio Access Technology Selection Algorithms for LTE-WiFi Network

TL;DR

This paper proposes optimal algorithms for radio access technology selection in LTE-WiFi HetNets to maximize throughput and minimize voice user blocking, validated through simulations.

Contribution

It introduces threshold-based optimal policies and algorithms for user association and offloading in LTE-WiFi HetNets, enhancing system throughput and reducing blocking.

Findings

Algorithms improve total system throughput

Reduced voice user blocking probability

Outperforms benchmark algorithms in simulations

Abstract

A Heterogeneous Network (HetNet) comprises of multiple Radio Access Technologies (RATs) allowing a user to associate with a specific RAT and steer to other RATs in a seamless manner. To cope up with the unprecedented growth of data traffic, mobile data can be offloaded to Wireless Fidelity (WiFi) in a Long Term Evolution (LTE) based HetNet. In this paper, an optimal RAT selection problem is considered to maximize the total system throughput in an LTE-WiFi system with offload capability. Another formulation is also developed where maximizing the total system throughput is subject to a constraint on the voice user blocking probability. It is proved that the optimal policies for the association and offloading of voice/data users contain threshold structures. Based on the threshold structures, we propose algorithms for the association and offloading of users in LTE-WiFi HetNet. Simulation results are presented to demonstrate the voice user blocking probability and the total system throughput performance of the proposed algorithms in comparison to another benchmark algorithm.

Figures

Tables

Table 1. Table I : Transition Probability Table.

$({𝑬}_{𝒍},𝒂)$	$({𝒊}^{\prime },{𝒋}^{\prime },{𝒌}^{\prime })$
$(\text{Arrival},A_1)$	$(i,j,k)$
$(\text{Voice departure from LTE},A_1)$	$(i-1,j,k)$
$(\text{Data departure from LTE},A_1)$	$(i,j-1,k)$
$(\text{Data departure from WiFi},A_1)$	$(i,j,k-1)$
$(\text{Voice arrival},A_2)$	$(i+1,j,k)$
$(\text{Data arrival},A_2)$	$(i,j+1,k)$
$(\text{Data arrival},A_3)$	$(i,j,k+1)$
$(\text{Voice arrival},A_4)$	$(i+1,j-1,k+1)$
$(\text{Voice departure from LTE},A_5)$	$(i-1,j+1,k-1)$
$(\text{Data departure from LTE},A_5)$	$(i,j,k-1)$
$(\text{Data departure from WiFi},A_5)$	$(i,j-1,k)$

Table 2. Table II : LTE Network Model.

Parameter	Value
Maximum voice capacity	$10$ users
Maximum data capacity	$10$ users
Voice bit rate of a single user	$20$ kbps
Data bit rate of a single user	$5$ Mbps
Voice packet payload	$50$ bits
Data packet payload	$600$ bits
Tx power for BS and MS	$46$ dBm and $23$ dBm
Noise figure for BS and MS	$5$ dB and $9$ dB
Antenna height for BS and MS	$32$ m and $1.5$ m
Antenna parameter for BS and MS	Isotropic antenna
Path loss	$128.1+37.6\log \left(R\right)$, $R$ in kms

Table 3. Table III : WiFi Network Model.

Parameter	Value
Channel bit rate	$54$ Mbps
UDP header	$224$ bits
Packet payload	$1500$ bytes
Slot duration	$20$ $\mu$s
Short inter-frame space (SIFS)	$10$ $\mu$s
Distributed Coordination Function IFS (DIFS)	$50$ $\mu$s
Minimum acceptable per-user throughput	$3.5$ Mbps
Tx power for AP	$23$ dBm
Noise figure for AP	$4$ dB
Antenna height for AP	$2.5$ m
Antenna parameter	Isotropic antenna

Equations

$$\begin{gathered} \mathcal{A}=\{\begin{array}{ll}{A}_1,& \text{parbox}[t]260.17464ptBlockthearrivinguserordonothing \\ duringdeparture,\\ A_2,& \text{Accept voice/data user in LTE,}\\ A_3,& \text{Accept data user in WiFi,}\\ A_4,& \text{parbox}[t]260.17464ptAcceptvoiceuserinLTEandoffload \\ onedatausertoWiFi,\\ A_5,& \text{parbox}[t]260.17464ptMoveonedatausertoaRAT(from \\ whichdeparturehasoccurred).\end{array} \end{gathered}$$

$${\mathcal{S}}_{E_1,a}=\{\begin{array}{ll}\mathcal{S}\setminus \{(0,0,0)\},& a=A_1,\\ \mathcal{S}\setminus \{(i,j,k):(i+j)=C\},& a=A_2,\\ \mathcal{S}\setminus \{(i,j,k):(j=0)||(k=W)\},& a=A_4,\\ \{\varnothing \},& \text{else}.\end{array}$$

$${\mathcal{S}}_{E_2,a}=\{\begin{array}{ll}\{(i,j,W):(i+j)=C\},& a=A_1,\\ \mathcal{S}\setminus \{(i,j,k):(i+j)=C\},& a=A_2,\\ \mathcal{S}\setminus \{(i,j,k):k=W\},& a=A_3,\\ \{\varnothing \},& \text{else}.\end{array}$$

$${\mathcal{S}}_{E_3,a}=\{\begin{array}{ll}\mathcal{S}\setminus \{(i,j,k):i=0\},& a=A_1,\\ \mathcal{S}\setminus \{(i,j,k):(i=0)||(k=0)\},& a=A_5,\\ \{\varnothing \},& \text{else}.\end{array}$$

$${\mathcal{S}}_{E_4,a}=\{\begin{array}{ll}\mathcal{S}\setminus \{(i,j,k):j=0\},& a=A_1,\\ \mathcal{S}\setminus \{(i,j,k):(j=0)||(k=0)\},& a=A_5,\\ \{\varnothing \},& \text{else}.\end{array}$$

$${\mathcal{S}}_{E_5,a}=\{\begin{array}{ll}\mathcal{S}\setminus \{(i,j,k):k=0\},& a=A_1,\\ \mathcal{S}\setminus \{(i,j,k):(j=0)||(k=0)\},& a=A_5,\\ \{\varnothing \},& \text{else}.\end{array}$$

$$r(s,E_2,A_2)=i{R}_{L,V}+(j+1)R_{L,D}+k{R}_{W,D}(k).$$

$$c(s,E_l,a)=\{\begin{array}{ll}1,& \text{if voice user is blocked},\\ 0,& \text{else}.\end{array}$$

$$\text{Maximize:}{V}^M=\underset{t\to \infty }{\lim }\frac{1}{t}{\mathbb{E}}_M[R(t)],$$

$$\begin{array}{rl}& \text{Maximize:}{V}^M=\underset{t\to \infty }{\lim }\frac{1}{t}{\mathbb{E}}_M[R(t)],\\ & \text{Subject to:}{B}^M=\underset{t\to \infty }{\lim }\frac{1}{t}{\mathbb{E}}_M[C(t)]\le B_{\max },\end{array}$$

$$\hat{r}(s,E_l,a)=r(s,E_l,a)\text{and}\hat{c}(s,E_l,a)=c(s,E_l,a).$$

$$\hat{r}(s,E_l,a;\beta )=\hat{r}(s,E_l,a)-\beta \hat{c}(s,E_l,a).$$

$$\begin{split}V(s)=&\sum\limits_{E_{l}}\hat{p}(s,E_{l})\max\limits_{a}[\hat{r}(s,E_{l},a;\beta)+V(s^{\prime})]+\big{(}1-\sum\limits_{E_{l}}\hat{p}(s,E_{l})\big{)}V(s),\end{split}$$

$${\beta }_{k+1}={\beta }_k+\frac{1}{k}(B^{{\pi }_{{\beta }_k}}-B_{\max }).$$

$$\begin{split}V_{n+1}(s)=&\sum\limits_{E_{l}}\hat{p}(s,E_{l})\max\limits_{a}[\hat{r}(s,E_{l},a;\beta)+V_{n}(s^{\prime})]+\big{(}1-\sum\limits_{E_{l}}\hat{p}(s,E_{l})\big{)}V_{n}(s),\end{split}$$

$$p{B}_{{\beta }^{\ast }-ϵ}+(1-p)B_{{\beta }^{\ast }+ϵ}=B_{\max }.$$

$$v(i,j,k)={\lambda }_v+{\lambda }_d+i{\mu }_v+j{\mu }_d+k{\mu }_d.$$

$$V^{{\pi }^{\ast }}(S_1)-V^{\pi }(S_1)=R_{L,D}-R_{W,D}(1).$$

$$V^{\pi^{*}}(S_{1})-V^{\pi}(S_{1})=\big{(}(k+1)R_{W,D}(k+1)-kR_{W,D}(k)-R_{L,D}\big{)}.$$

$$\begin{split}&V(i,j,k)=\lambda_{v}\delta\max\{f(i,j,k)-\beta+V(i,j,k),f(i+1,j,k)+V(i+1,j,k)\}+\lambda_{d}\delta\big{(}f(i,j+1,k)+V(i,j+1,k)\big{)}\\ &+i\mu_{v}\delta\big{(}f(i-1,j+1,k-1)+V(i-1,j+1,k-1)\big{)}+j\mu_{d}\delta\big{(}f(i,j,k-1)+V(i,j,k-1)\big{)}\\ &+k\mu_{d}\delta\big{(}f(i,j,k-1)+V(i,j,k-1)\big{)}+\big{(}1-v(i,j,k)\big{)}V(i,j,k).\end{split}$$

$$V_{1,n}(i,j,k,a)=\{\begin{array}{ll}-\beta +V_{n-1}(i,j,k),& a=A_1,\\ R_{L,V}+V_{n-1}(i+1,j,k),& a=A_2.\end{array}$$

$$V_{1,n}(i,j,k)=\underset{a\in \{A_1,A_2\}}{\max }{V}_{1,n}(i,j,k,a).$$

$$V^1(i,j,k)=\underset{n\to \infty }{\lim }{V}_{1,n}(i,j,k).$$

$$D_i{V}_{1,n}(i,j,k,a)=\{\begin{array}{ll}{D}_i{V}_{n-1}(i,j,k),& a=A_1,\\ D_i{V}_{n-1}(i+1,j,k),& a=A_2.\end{array}$$

$$D_{ii}{V}_{1,n}(i,j,k,a)=\{\begin{array}{ll}{D}_{ii}{V}_{n-1}(i,j,k),& a=A_1,\\ D_{ii}{V}_{n-1}(i+1,j,k),& a=A_2.\end{array}$$

$$\begin{array}{rl}& 2V_{1,n}(i+1,j,k)\ge V_{1,n}(i+1,j,k,a_1)+V_{1,n}(i+1,j,k,a_2)\\ & =V_{1,n}(i+2,j,k,a_1)+V_{1,n}(i,j,k,a_2)-D_i{V}_{1,n}(i+1,j,k,a_1)+D_i{V}_{1,n}(i,j,k,a_2).\end{array}$$

$$\begin{array}{rl}& X=D_i{V}_{n-1}(i,j,k)-D_i{V}_{n-1}(i+1,j,k)\\ & =-D_{ii}{V}_{n-1}(i,j,k)\ge 0.\end{array}$$

$$\begin{array}{r}X=D_i{V}_{n-1}(i+1,j,k)-D_i{V}_{n-1}(i+1,j,k)=0.\end{array}$$

$$\begin{array}{rl}& X=D_i{V}_{n-1}(i+1,j,k)-D_i{V}_{n-1}(i+2,j,k)\\ & =-D_{ii}{V}_{n-1}(i+1,j,k)\ge 0.\end{array}$$

$$\begin{array}{rl}& X=D_i{V}_{n-1}(i,j,k)-D_i{V}_{n-1}(i+2,j,k)\\ & =-D_{ii}{V}_{n-1}(i,j,k)-D_{ii}{V}_{n-1}(i+1,j,k)\ge 0.\end{array}$$

Full text

I Introduction

To meet the ever-increasing Quality of Service (QoS) requirements of users, various Radio Access Technologies (RATs) have been standardized [1]. Each RAT has different characteristics regarding associated parameters like coverage and capacity. It has been predicted that by $2021$ monthly global mobile data traffic will exceed $49$ exabytes [2]. This unprecedented growth in data traffic has become one of the serious challenges for cellular network operators. To address this issue, both from users’ and network providers’ point of view, it has become necessary that different RATs interwork with each other. A wireless network where different RATs are present, and users can be associated and moved seamlessly from one RAT to another, is called a Heterogeneous Network (HetNet). In this paper, our aim is to determine the optimal RAT selection policy in a HetNet 111The terminologies “RAT selection” and “association” has been used interchangeably throughout the paper..

Due to the complementary characteristics of Third Generation Partnership Project (3GPP) Long Term Evolution (LTE) Base Stations (BSs) providing ubiquitous coverage and IEEE 802.11 [3] based Wireless Local Area Network (WLAN) (also known as Wireless Fidelity (WiFi)) Access Points (APs) providing high bit rate capability in hot-spot areas, interworking between them [4] offers an interesting solution. In areas where both LTE and WiFi coverage are present, a user can be associated with either of them. Moreover, data users can be steered from one RAT to another to achieve load balancing. This proposal, known as mobile data offloading, has been introduced in $3$GPP Release $12$ specifications [4]. Since WiFi operates in unlicensed spectrum and most of the commercially available user equipments already have a dedicated WLAN interface, this proposal has become popular both with network operators and handset manufacturers.

For efficient utilization of both LTE and WiFi networks, it is necessary to take appropriate association and offloading decisions. RAT selection and offloading decisions can be made either at the user side or the network side. In user-initiated RAT selection schemes, there is no cooperation between LTE and WiFi networks, and users decide which RAT should be selected based on certain criteria. Since users individually take selfish RAT selection decisions to maximize individual utility functions, this may not provide a globally optimum solution [5]-[8]. To address this issue, a network-initiated RAT selection algorithm, which optimizes different network parameters, becomes necessary.

In this paper, we investigate an optimal association policy for an LTE-WiFi HetNet, as illustrated in Fig.1. Network-initiated RAT selection and offloading decisions are taken by a centralized controller possessing an overall view of the network. We consider two types of users, viz., voice and data users, to be present inside the LTE-WiFi HetNet. We consider that voice users are always associated with LTE since unlike LTE, WiFi may not provide the required QoS for a voice user. However, data users can be associated with either LTE or WiFi. Offloading of data users from one RAT to another is considered at the time of association of voice users or departure of existing voice/data users. From a network operator’s perspective, total system throughput is an important system metric since the generated revenue may largely depend on the number of bytes transported by the operator. Moreover, data users experiencing high throughput are more likely to adhere to a network operator, thus facilitating the improvement of the customer base of the operator. Therefore, we aim to maximize the total system throughput and formulate this as a continuous-time Markov Decision Process (MDP) problem.

In the case of data users, although in low load condition, WiFi usually provides higher throughput than that of LTE, as the WiFi load increases, average per-user throughput in WiFi decreases rapidly [9]. Therefore, under high WiFi load, LTE may offer more throughput than WiFi to data users and thus may be preferable to data users for the association. However, voice and data users are allocated resources in LTE from a common resource block pool. The throughput requirement of LTE data users is usually more than that of the voice users. Therefore, maximization of the total system throughput may result in excessive blocking of voice users. The system may attempt to save LTE resources which can be allocated later to data users having greater contributions to the system throughput than that of voice users. It results in an inherent trade-off between the total system throughput and the blocking probability of voice users. We consider this problem within the formalism of Constrained Markov Decision Process (CMDP), which maximizes the total system throughput subject to a constraint on the voice user blocking probability.

It is proved that the associated optimal policies contain a threshold structure, where after a certain threshold on the number of WiFi data users, data users are served using LTE. The existence of a similar threshold for the blocking of voice users is also established. Based on the threshold based optimal policy, we propose two RAT selection algorithms for LTE-WiFi HetNet. Extensive simulations are performed in ns-$3$ (a discrete event network simulator) [10] to evaluate the performance of the proposed association algorithms. Using simulation results, performance gains of the proposed algorithms in comparison to another algorithm in the literature [11] are also evaluated.

I-A Related Work

The solutions which investigate RAT selection problem in a HetNet, can be broadly divided into two categories, viz., user-initiated [5]-[8] and network-initiated [12]-[22]. In [5], a user-initiated RAT selection algorithm based on Signal-to-Noise Ratio (SNR) and load information of individual RATs with the adaptation of hysteresis mechanism, is considered for LTE-WiFi HetNet. The performance of this scheme is compared with network-initiated cell-range extension schemes that use network-optimized Received Signal Strength Indicator (RSSI) bias value to steer users to other RATs. In [6], a distributed RAT selection algorithm is proposed based on the distance and peak rate obtained from different IEEE 802.11 [3] APs.

Few heuristic-based network-initiated RAT selection approaches are considered in [18]-[20]. While the algorithm proposed in [19] prefers WLAN over cellular regardless of the service type, the one proposed in [20] prefers cellular RAT for voice users and WLAN for data users. Among the other network-initiated RAT selection schemes, [12]-[17], [21]-[22] consider various optimization approaches. In [12], optimal RAT selection problem is addressed in a HetNet to optimize throughput, blocking probability, etc.. Since the associated algorithm scales exponentially with the system size, authors also propose a computationally efficient heuristic policy. In [13], the association resulting in maximum value for the sum of logarithms of throughputs is chosen as the optimal association among Wireless Stations (STAs) and APs. However, authors do not take into account user arrival and departure. RAT selection policies in wireless networks [23]-[25] are sometimes observed to contain certain threshold structures. A multi-class admission system is considered in [23], where it is demonstrated that if it is optimal to accept a user of a class, then it is optimal to accept a user of higher profit class too.

Offloading of data users from one RAT to another plays a major role in the capacity improvement of the system. Performance improvement achieved by on-the-spot offloading [26], a user-initiated WiFi offloading scheme, is analyzed in [11]. The basic idea behind on-the-spot offloading is to steer the mobile data users to WiFi, whenever WiFi is available. The user-initiated offloading scheme in [27] is based on the combined information of signal strength and network load of LTE/WLAN. However, being a greedy one, this algorithm fails to converge to a globally optimum solution. The network-initiated offloading approach in [28] computes the optimal fraction of traffic to be offloaded to WiFi such that the per-user throughput of the system is maximized and performs better than on-the-spot offloading [11]. However, the model in [28] does not incorporate voice users inside an LTE network.

I-B Our Contribution

In this paper, we investigate the optimal association policy in an LTE-WiFi HetNet. We consider a system where voice and data users can arrive or depart at any point in time. We introduce the possibility of data user offloading from one RAT to another at the time of association or departure of a user. We target to maximize the total system throughput. The problem is formulated within the framework of MDP. Another formulation is developed where we target to maximize the total system throughput, subject to a constraint on the voice user blocking probability, using CMDP. Threshold structures of optimal policies are established. We propose two algorithms based on the computed optimal policies and implement in ns-$3$. 3GPP recommended parameters are used in the simulations. Since most of the practical offloading schemes offload data users to WiFi, performances of the proposed algorithms are compared with on-the-spot offloading algorithm [11].

The arrival of a new user in the LTE-WiFi system triggers the need for the optimal RAT selection. Also, with the arrival or departure of users, the active users in different RATs may need to get offloaded to other RATs. While few works in the literature have focused on RAT selection and offloading techniques, respectively, no existing literature, to the best of our knowledge, has addressed the issue of joint RAT selection and offloading for LTE-WiFi HetNet.

The rest of the paper is organized as follows. The system model is described in Section II. In Section III, the RAT selection problems are formulated within the framework of unconstrained and constrained continuous-time MDP, respectively. In Section IV, we derive the threshold structure of the optimal policy. Algorithms for the association of voice and data users in LTE-WiFi HetNet are proposed in Section V. Section VI presents simulation results. In Section VII, we conclude the paper.

II System Model

We consider a system where an LTE BS and a WiFi AP are present. As illustrated in Fig.1, we assume that both the BS and the AP are connected to a centralized controller by lossless links. We assume that the voice and data users are geographically located at any point in the LTE BS coverage area. Since data users outside the dual coverage area of the LTE BS and the WiFi AP always get associated with the LTE BS and no decision is involved in this case, without loss of generality, we take into consideration only those data users which are present inside the WiFi AP coverage area. We assume that there is a common resource pool in LTE for the voice users as well as the data users inside the WiFi AP coverage area. Data users inside the dual coverage area can be associated with the LTE BS or the WiFi AP. All the users are assumed to be stationary. Voice and data user arrivals follow Poisson processes with means $\lambda_{v}$ and $\lambda_{d}$, respectively. Service times for voice and data user are exponentially distributed with means $\frac{1}{\mu_{v}}$ and $\frac{1}{\mu_{d}}$, respectively. For justification behind these assumptions, see [29].

Remark 1**.**

*Although for brevity of notation, a single LTE BS and a single AP have been considered, the system model can be generalized to a single LTE BS and multiple APs with non-overlapping coverage areas. Moreover, considering that each point in a geographical area is mapped to a single LTE BS (the LTE BS with highest average signal strength, say), multiple BSs can also be included in the system model with slight modifications. *

II-A State Space

We model the system as a controlled continuous time stochastic process $\{X(t)\}_{t\geq 0}$ defined on a state space $\mathcal{S}$. Any state $s\in\mathcal{S}$ is represented as a $3$-tuple ${s}=(i,j,k),$ where $i,j$ and $k$ represent the number of voice users in LTE, the number of data users in LTE and the number of data users in WiFi, respectively. The system state remains unchanged unless an existing user departs or a new user arrives in the system. The arrivals and departures in the system are referred to as events. Five types of events are possible, viz., ($E_{1}$) an arrival of a new voice user in the system, ($E_{2}$) an arrival of a new data user in the system, ($E_{3}$) a departure of an existing voice user from LTE, ($E_{4}$) a departure of an existing data user from LTE and ($E_{5}$) a departure of an existing data user from WiFi. Whenever an event occurs, the centralized controller takes an action, and based on the type of event and the action taken by the controller, a state transition may happen. Note that the transitions of $\{X(t)\}_{t\geq 0}$ happen only at event epochs and not otherwise. Thus, it suffices to observe the system state only at event epochs. A finite amount of reward and cost are associated with every state-action pair. Detailed descriptions of the action space, state transitions, reward and cost are provided in subsequent subsections.

Next, we elaborate on the structure of $\mathcal{S}$. We assume that $(i,j,k)\in\mathcal{S}$ if $(i+j)\leq C$ and $k\leq W,$ where $C$ is the total number of common resource blocks reserved in LTE for voice and data users, and $W$ is the maximum number of users in WiFi, so that the per-user throughput in WiFi is greater than a threshold. The condition $(i+j)\leq C$ arises because we assume that in each LTE subframe, every admitted user is allocated one resource block. If this allocation is not possible, a new user is not admitted in the LTE system. Furthermore, note that WiFi throughput decays monotonically [9] as the number of WiFi users increases. We assume that each user gets more than a threshold value of average throughput (say $2$ Mbps), which leads to the bound $W$ on the maximum number of users that can be accommodated in the WiFi system.

Remark 2**.**

Although the allocation of multiple resource blocks is closer to the practical scenario, this complicates the system model while the methodology and approach adopted in this paper do not change.

II-B Action Space

The set of actions defines a set of possible association and offloading strategies in the event of arrival or departure of a user. Let the action space be denoted by $\mathcal{A}$. Depending on the arrival or departure, we have a set of actions as stated below.

[TABLE]

Remark 3**.**

In this paper, actions are chosen based on the system state and the event occurred. One way of representing this is embedding the event in the state space so that the action depends only on the system state. However, to avoid notational complications associated with this approach, we view the action as a function of the system state and the event.

Let the set of states (subset of $\mathcal{S}$) in which action $a$ chosen based on an event $E_{l}$ is feasible be denoted by ${\mathcal{S}}_{E_{l},a}$. Thus, in the case of voice user arrival, we have,

[TABLE]

For data user arrival,

[TABLE]

For voice user departure from LTE,

[TABLE]

For data user departure from LTE,

[TABLE]

For data user departure from WiFi,

[TABLE]

In the case of voice and data user arrivals, the set of all possible actions are $\{A_{1},A_{2},A_{4}\}$ and $\{A_{1},A_{2},A_{3}\}$, respectively. However, when an event $E_{l}$ occurs, action $a$ is not feasible if the system state is not present in ${\mathcal{S}}_{E_{l},a}$. In this paper, voice user blocking ($A_{1}$) is considered to be a feasible action in all the states, provided the system is not empty. We consider blocking as a feasible action for data users, only when capacity is reached for both the RATs. When a user departs from LTE or WiFi, the controller can choose either $A_{1}$ or $A_{5}$. If after the departure of a user from LTE, $A_{5}$ is chosen, it offloads one data user from WiFi to LTE.

II-C State Transitions

Based on an event and an action chosen, from a state, the system moves deterministically to a different state. Assume that from the state $s=(i,j,k)$, the system moves to the state $s^{\prime}(E_{l},a)=(i^{\prime},j^{\prime},k^{\prime})$ under the event $E_{l}$ and chosen action $a$. Values of $i^{\prime},j^{\prime}$ and $k^{\prime}$ for different events $E_{l}$ (arrivals and departures of users) and action $a$ are tabulated in Table I.

Note that this table is exhaustive in all kinds of events and actions. However, in a state, we need to consider only those events and actions which are feasible in that state.

II-D Rewards and Costs

Let the reward and cost functions per unit time corresponding to a state $s$, event $E_{l}$ and action $a$ be represented by $r(s,E_{l},a)$ and $c(s,E_{l},a)$, respectively. Let $R_{L,V}$ and $R_{L,D}$ denote the bit rate of voice and data users in LTE, respectively. To keep the model simple and computationally tractable, we assume that the bit rate of data users (e.g. data services like interactive video conferencing) in LTE is constant. In general, a voice user generates constant bit rate (CBR) traffic, and hence we take $R_{L,V}$ to be a constant. $R_{W,D}(k)$ corresponds to the per-user data throughput of $k$ users in WiFi. We assume full buffer traffic model [9] for WiFi. The calculation of $R_{W,D}(k)$ is based on the contention-driven medium access of WiFi users. It is a function of the probabilistic transmission attempts of the users, corresponding success and collision probabilities, and slot times for successful transmission, idle slots and busy slots during collisions. The reward per unit time in a state under the occurrence of an event and an action chosen is defined as the total system throughput in that state under that event and the chosen action. For example, in the case of data user arrivals and $A_{2}$, it can be expressed as

[TABLE]

The cost function considered here is as follows. Whenever the centralized controller blocks an incoming voice user, one unit cost is incurred per unit time. Otherwise it is zero. Thus,

[TABLE]

We consider blocking of data users only when both LTE and WiFi systems are full. Hence, we do not consider any cost on the blocking of data users.

III Problem Formulation And Solution Methodology

A decision rule describes the mapping regarding which action is to be chosen at different states $s\in S$ and decision epochs $t_{n}$. An association policy is a sequence of decision rules $({\pi}^{t_{1}},{\pi}^{t_{2}},\ldots,{\pi}^{t_{n}},\ldots)$ taken at different decision epochs. Our goal is to determine an association policy which maximizes the total system throughput. This can be formulated as a continuous-time unconstrained MDP problem. In this case, a pure optimal policy exists [30]. Since the contribution of data users to the total system throughput is more than that of the voice users, the optimal association policy may result in high blocking probability of voice users. Hence, to address the trade-off between the total system throughput and the voice user blocking probability, we consider the CMDP problem, where we target to maximize the total system throughput, subject to a constraint on the voice user blocking probability. In this case, a stationary randomized optimal policy exists [31]. A Randomized policy is a mixture of two pure policies with associated probabilities. Arrival and departure of users can occur at arbitrary points in time, which makes the problem continuous time in nature.

III-A Problem Formulation

Let $\mathcal{M}$ be the set of all memoryless policies. To guarantee a unique stationary distribution, we assume that Markov chains associated with such policies are irreducible. Following the policy $M\in\mathcal{M}$, let the average reward and cost of the system over infinite horizon be denoted by $V^{M}$ and $B^{M}$, respectively. Let $R(t)$ and $C(t)$ be the total reward and cost of the system incurred up to time $t$, respectively. For the unconstrained MDP problem, our objective is to maximize the total system throughput which can be described as follows.

[TABLE]

where $\mathbb{E}_{M}$ denotes the expectation operator under the policy $M$. However, for the CMDP problem, our objective is to maximize the total system throughput, subject to a constraint on the blocking probability of voice users. This can be described as follows.

[TABLE]

where $B_{\max}$ denotes the constraint on the blocking probability of voice users. Our objective is to determine the optimal policy for both unconstrained and constrained MDP problem. Since the optimal policies are known to be stationary policies, the corresponding limits in Equation (1) and (2) exist.

III-B Conversion to Discrete-Time MDP

To compute the optimal policy, we can use the well-known Value Iteration Algorithm (VIA) [30]. However, before that, the continuous-time MDP has to be transformed into an equivalent discrete-time MDP using uniformization [30], so that both models have the same average expected reward and cost for a stationary policy.

Let $\tau_{s}(E_{l},a)$ represent the expected time until the next event, if action $a$ is chosen in state $s$ under the event $E_{l}$. We need to choose a number $\delta$, such that $0<\delta\ll\displaystyle\min_{s,E_{l},a}\tau_{s}(E_{l},a)$. The state space and the action space remain the same in the equivalent discrete-time model. Let $\hat{p}(s,E_{l})$, $\hat{r}(s,E_{l},a)$ and $\hat{c}(s,E_{l},a)$ represent the probabilities of the event, reward and cost in the transformed discrete-time model in state $s$ under the event $E_{l}$ and action $a$, respectively. Thus, we have,

[TABLE]

$\hat{p}(s,E_{l})$ is a function of rate of different events $E_{l}$ and $\delta$. Note that, this discrete-time MDP has identical stationary policies to that of the continuous-time MDP.

III-C Lagrangian Approach

After conversion into an equivalent discrete-time MDP model, we use the Lagrangian approach [31] to solve the CMDP. For a fixed value of Lagrange Multiplier (LM) $\beta$, the modified reward function of the CMDP is

[TABLE]

The dynamic programming equation below describes the necessary condition for optimality.

[TABLE]

where $V(s)$ denotes the value function in state $s\in\mathcal{S}$. Next, our aim is to determine the value of $\beta$ ($=\beta^{*}$, say) which maximizes the average expected reward, subject to a cost constraint. The value of $\beta^{*}$ can be determined using gradient descent algorithm, as discussed in [32]. In $k$th iteration, we modify the value of $\beta$ from its previous iteration as,

[TABLE]

For a fixed value of $\beta$, the unconstrained maximization problem can be solved using VIA, as described below.

[TABLE]

where $V_{n}(.)$ is an estimate of the value function after $n$th iteration. After determining $\beta^{*}$, the next step is to determine the optimal policy for the CMDP problem. As discussed in [31], optimal policy for a CMDP problem is a mixture of two pure policies $\pi_{\beta^{*}-\epsilon}$ and $\pi_{\beta^{*}+\epsilon}$, obtained by perturbation of $\beta^{*}$ by a small amount $\epsilon$ in both directions. Let the long-term average expected costs of the two pure policies be $B_{\beta^{*}-\epsilon}$ and $B_{\beta^{*}+\epsilon}$, respectively. In the next step, we determine the value of the parameter $p$ such that

[TABLE]

Finally, we have a randomized optimal policy for the considered CMDP problem. At each decision epoch, policies $\pi_{\beta^{*}-\epsilon}$ and $\pi_{\beta^{*}+\epsilon}$ are chosen w.p. $p$ and $(1-p)$, respectively.

Note that the iterations on LM described above are necessary only for the CMDP problem. In the case of unconstrained MDP, VIA can be employed to determine a pure optimal policy, after an equivalent discrete-time MDP model is obtained.

IV Structure of the Optimal Policy

The dynamic programming equations (Equation (3) and (5)) described in the previous section are exploited to establish the fact that the optimal policy is of threshold type. The optimality of threshold policy is established with the aid of some lemmas presented below. For the purpose of readability, we present the proofs of the lemmas in Appendices.

IV-A Optimal Policy for Data Users

In this section, we present structural properties on the optimal policy for the service of data users along with their physical interpretations. Let us denote the throughput increment in WiFi when the number of WiFi users increases from $k$ to $(k+1)$ by $\tilde{R}_{W,D}(k)$. Therefore, $\tilde{R}_{W,D}(k)=(k+1){R}_{W,D}(k+1)-k{R}_{W,D}(k)$. We assume the following.

Assumption 1**.**

Let $R_{L,D}$ be such that $R_{L,D}\geq\tilde{R}_{W,D}(k)$, $\forall k\geq k_{th}$ and $R_{L,D}<\tilde{R}_{W,D}(k),\forall k<k_{th},$ where $k_{th}$ is a threshold such that if $k\geq k_{th}$, the data rate improvement provided by a single data user in the LTE system is more than the improvement in total WiFi throughput as the number of WiFi data users is increased from $k$ to $(k+1)$.

Remark 4**.**

Following the full buffer traffic model [9], $\tilde{R}_{W,D}(k)$ initially increases with $k$ and then decreases. This behavior matches with Assumption 1.

The following two lemmas describe a threshold structure on the optimal policy for the service of data users. Specifically, up to a certain threshold on the total number of data users, data users are served using WiFi. After the threshold is crossed, data users are served using LTE.

Lemma 1**.**

For every $i$ and $j$ such that $(i+j)<C$, if the total number of data users in the system is $(j+k)\leq k_{th}$, then the optimal policy is to serve all data users using WiFi. In other words,$(j+k)\leq k_{th}\implies j=0$.

Proof.

See Apppedix A. ∎

In Lemma 1, following Assumption 1, since for $k<k_{th}$, the data rate improvement is more if an additional data user is served using WiFi rather than using LTE, it is optimal to serve the data users using WiFi.

Lemma 2**.**

For every $i$ and $j$ such that $(i+j)<C$, if the total number of data users in the system is $(j+k)>k_{th}$, then the optimal policy is to serve $k_{th}$ data users using WiFi and all other data users using LTE. In other words,$(j+k)>k_{th}\implies k=k_{th}$.

Proof.

See Apppedix B. ∎

The physical significance of Lemma 2 is that for $k\geq k_{th}$, the data rate improvement provided by a single data user in LTE is more than that of the WiFi (following Assumption 1), and hence it is optimal to serve up to $k_{th}$ data users using WiFi and serve the additional data users using LTE.

Following lemma is a direct consequence of how the system is modeled.

Lemma 3**.**

For every $i$ and $j$ such that $(i+j)=C$, the optimal policy is to serve all the incoming data users using WiFi until $k=W$, where an incoming data user is blocked.

IV-B Optimal Policy for Voice Users

In this section, we characterize the optimal policy for the arrival of voice users. We prove that the optimal policy is of threshold type. Let $D_{i}$ be the difference operator which is defined as $D_{i}V(i,j,k)=V(i+1,j,k)-V(i,j,k)$. Similarly, we define the second difference operator as $D_{ii}(.)=D_{i}(D_{i}(.))$. Let $E_{i}$ be another difference operator defined as $E_{i}V(i,j,k)=V(i+1,j-1,k+1)-V(i,j,k)$. We define the second difference operator as $E_{ii}(.)=E_{i}(E_{i}(.))$. Similarly, we define $F_{i}V(i,j,k)=V(i+1,j-1,k)-V(i,j,k)$. In this section, the terminologies “increasing” and “decreasing” are used in the weak sense of “non-decreasing” and “non-increasing”, respectively. In each state, let the sum of arrival and service rates be denoted by $v(i,j,k)$. Thus, we have,

[TABLE]

Let us define $f(i,j,k)=(iR_{L,V}+jR_{L,D}+kR_{W,D}(k))$. The lemma presented below describes the superiority of one action over the other for the service of incoming voice users. Specifically, up to a certain threshold on the total number of data users, $A_{4}$ (accept voice user in LTE with data user offload to WiFi) is better than $A_{2}$ (accept voice user in LTE). After the threshold is crossed, $A_{2}$ becomes better.

Lemma 4**.**

In the case of a voice user arrival in a state $(i,j,k)$, where $(i+j)<C$,

• (i)

$A_{4}$* is always better than $A_{2}$ if $k<k_{th}$,*

• (ii)

$A_{2}$* is always better than $A_{4}$ if $k\geq k_{th}$.*

Proof.

Proof is similar to the proof of Lemma 1. ∎

Similar to Lemma 1 and 2, following Assumption 1, since for $k<k_{th}$, the data rate improvement is more if an additional data user is served using WiFi rather than using LTE, $A_{4}$ is better than $A_{2}$. Therefore, when $k<k_{th}$, the choice of optimal action is between $A_{4}$ (accept voice user in LTE with data user offload to WiFi) and $A_{1}$ (blocking). Similarly, for $k\geq k_{th}$, the optimal action is either $A_{2}$ (accept voice user in LTE) or $A_{1}$.

The following lemma describes that when capacity is not reached in LTE and a voice user arrives, a threshold structure is observed. Until a threshold on the number of voice/data users in LTE, $A_{2}$ (for $k\geq k_{th}$) or $A_{4}$ (for $k<k_{th}$) is preferred. After the threshold $A_{1}$ becomes optimal.

Lemma 5**.**

For every $i$ and $j$ such that $(i+j)<C$ and a voice user arrival,

• (i)

if the optimal action in state $(i,j,k)$ is $A_{1}$, then the optimal action in state $(i+1,j,k)$ and in state $(i,j+1,k)$ is also $A_{1}$,

• (ii)

if the optimal action in state $(i,j,k)$ is $A_{2}$ ($A_{4}$), then the optimal action in state $(i-1,j,k)$ and in state $(i,j-1,k)$ is also $A_{2}$ ($A_{4}$*). *

Proof.

See Apppedix C. ∎

When the number of voice/data users in LTE is less, $A_{2}$ or $A_{4}$ is chosen as the optimal action in the event of a voice user arrival. When $i$ or $j$ crosses a certain threshold, the number of free resources for incoming voice users decreases. Therefore, the blocking probability of voice users increases. Thus, after a threshold on $i$ or $j$, $A_{1}$ becomes optimal.

However, when $(i+j)=C$, since $A_{2}$ is infeasible, optimal action is either $A_{1}$ or $A_{4}$. The lemma presented below discusses the threshold nature of the optimal policy for voice user arrivals when $(i+j)=C$.

Lemma 6**.**

For every $i$ and $j$ such that $(i+j)=C$ and a voice user arrival,

• (i)

if the optimal action in state $(i,j,k)$ is $A_{1}$, then the optimal action in state $(i+1,j-1,k)$ is also $A_{1}$,

• (ii)

if the optimal action in state $(i,j,k)$ is $A_{4}$, then the optimal action in state $(i-1,j+1,k)$ is also $A_{4}$.

Proof.

See Apppedix D. ∎

The physical interpretation of the above lemma is that for states with $(i+j)=C$, when $i$ is small, $A_{4}$ is preferred. However, when $i$ crosses a threshold, since $j$ becomes small, and consequently the total system throughput is small, $A_{4}$ may further lower the total system throughput. Therefore, blocking of voice users is chosen as the optimal action.

V Proposed Network-Initiated Association Algorithm

Based on the optimal policy computed by solving the unconstrained MDP and CMDP problem, respectively, in this section, we propose two network-initiated association algorithms for LTE-WiFi HetNet. The details of the unconstrained MDP-based algorithm is presented below.

The procedure CALC–OPT–POLICY–UC in Algorithm 1 computes the optimal policy by solving an unconstrained MDP problem using solution methodologies described in Section III. The calculated thresholds for the association of voice/data users are made available to the centralized controller connected to both the LTE BS and the WiFi AP. Since the centralized controller has an overall view of the whole system, the information regarding the numbers of active voice and data users in LTE and WiFi, are available to it. Whenever there is an arrival or a departure, the controller initiates the procedure POLICY–IMPL, as described in Algorithm 1. This procedure determines the state of the system based on the number of active users in LTE and WiFi networks and then chooses the appropriate action based on the corresponding thresholds.

Next, we describe the CMDP-based algorithm which addresses the issue of high blocking probability of voice users, which may be encountered in Algorithm 1. The algorithm is described in detail below.

Apart from the same set of input parameters as required by Algorithm 1, Algorithm 2 requires $B_{\max}$ as an additional parameter to specify the constraint on the blocking probability of voice users. The procedure CALC–OPT–POLICY in Algorithm 2 computes the randomized optimal policy for the considered CMDP problem. First, the optimal policy is determined for a fixed value of $\beta$, and then the value of $\beta$ is updated until $B^{\pi_{\beta}}$ becomes equal to $B_{\max}$. All other procedures are similar to the procedures described in Algorithm 1.

VI Numerical and Simulation Results

In this section, the algorithms proposed in the last section are implemented in ns-$3$ to observe the performance of the proposed algorithms. Performance of the proposed algorithms in terms of blocking probability of voice users and the total system throughput is compared to the performance of on-the-spot WiFi offloading algorithm [11]. In this algorithm [11], data user chooses LTE, only when there is no WiFi coverage. Therefore, in the considered system model, with on-the-spot offloading, data users always get associated with WiFi until WiFi capacity is exhausted. Voice users always get associated with LTE BS, and when LTE capacity is full, they are blocked.

VI-A Simulation Model and Evaluation Procedure

The simulated network model consists of a $3$GPP LTE BS and an IEEE $802.11$g WiFi AP. All users are taken to be stationary. The AP is approximately $50$ m away from the LTE BS, and data users are distributed uniformly within $30$ m radius of the WiFi AP. The WiFi AP is assumed to be deployed by the same cellular operator and hence trusted from the point of view of interworking. LTE and WiFi network parameters used in the simulation, as summarized in Table II and III, are based on 3GPP models [33]-[34] and saturation throughput [9] $802.11$g WiFi model. Propagation delay in WiFi network is assumed to be negligible. We consider CBR traffic for voice and data users in LTE. The generation of a fixed rate uplink flow is implemented in ns-$3$ using an application developed by us, which works similar to the ON/OFF application. This application creates sockets between the sender and the receiver, and fixed sized packets are transmitted from the sender to the receiver at a constant bit rate.

VI-B Voice User Arrival Rate Variation

VI-B1 Voice User Blocking Probability Performance

Fig. 2(a) illustrates the variation of voice user blocking percentage of on-the-spot offloading [11], Algorithm 1 and 2 as a function of $\lambda_{v}$.

In on-the-spot offloading, voice users are blocked when LTE reaches the capacity. When $\lambda_{v}$ is small, the voice user blocking probability is small. However, as $\lambda_{v}$ increases, the probability of approaching the LTE capacity and hence the voice user blocking probability increases. The voice user blocking probability in Algorithm 1 is small when $\lambda_{v}$ is small. However, as $\lambda_{v}$ increases, voice user blocking probability values become marginally higher than the corresponding values for on-the-spot offloading. Algorithm 1 may introduce blocking of voice users even when LTE has not reached its capacity, i.e., for states with $(i+j)<C$. Voice users have very less contribution to the total system throughput. Hence, voice users are blocked to save resources for data users which contribute significantly to the total system throughput. However, in Algorithm 2, the number of states with proactive blocking (blocking when $(i+j)<C$) is reduced due to a constraint on the voice user blocking probability. Additionally, when $i$ is small, the optimal action in states with $(i+j)=C$ becomes $A_{4}$ (accept voice user in LTE and data offload to WiFi). Voice user blocking probability contribution comes mainly from the states with $(i+j)=C$, where $i$ is large (say states $(C,0,0)$,$(C-1,1,0)$ etc.). Since a major fraction of voice user blocking occurs when $(i+j)=C$ and $i$ is large, the system becomes analogous to the on-the-spot offloading. Hence, the voice user blocking probability performance of Algorithm 2 is almost similar to on-the-spot offloading algorithm [11].

VI-B2 Total System Throughput Performance

Total system throughput performance comparison of different algorithms is illustrated in Fig. 2(b). In on-the-spot offloading, the average number of voice users in LTE increases with $\lambda_{v}$, while the average number of data users in WiFi remains constant. Thus, the total system throughput increases with $\lambda_{v}$. For Algorithm 1, with an increase in $\lambda_{v}$, the blocking probability of voice users increases. Therefore, the fraction of voice users in the system decreases, and the total system throughput increases. Besides, Algorithm 1 performs a significant amount of load balancing under $A_{4}$ (accept voice user in LTE with data user offload to WiFi) and $A_{5}$ (move data user to the RAT from where a user has departed). With higher $\lambda_{v}$, load balancing actions are chosen more frequently. Thus, with higher $\lambda_{v}$, Algorithm 1 exhibits greater improvement over on-the-spot offloading algorithm. The improvement in total system throughput varies from $1.22\%$ (for $\lambda_{v}=0.01$) to $10.32\%$ (for $\lambda_{v}=0.25$). In Fig. 2(b), we observe that Algorithm 2 also performs better than on-the-spot offloading. However, due to a constraint on the voice user blocking probability, performance improvement is lower than Algorithm 1. For lower values of $\lambda_{v}$ ($\lambda_{v}=0.01,0.07$), the total system throughput of Algorithm 2 is same as that of Algorithm 1 as the optimal policy for the CMDP is same as that of the unconstrained MDP. On-the-spot offloading algorithm blocks voice users only when LTE reaches capacity. Typically, in Algorithm 2 also, voice user blocking occurs when the LTE is full with a large number of voice users. However, due to load balancing of data users, Algorithm 2 outperforms the on-the-spot offloading algorithm. With $\lambda_{v}=0.01$, the improvement in total system throughput is only $1.22\%$ and with $\lambda_{v}=0.25$, it becomes $7.60\%$.

VI-C Data User Arrival Rate Variation

VI-C1 Voice User Blocking Probability Performance

In Fig. 2(c), for on-the-spot offloading, voice and data users are accepted in LTE and WiFi, respectively. Consequently, changes in $\lambda_{d}$ do not affect the blocking probability performance of voice users in LTE. In the case of Algorithm 1, increase in $\lambda_{d}$ associates more number of data users with LTE, since the optimal policy for data users is to associate with LTE after the number of WiFi data users crosses a certain threshold. Therefore, the number of free LTE resources for voice users reduces, eventually increasing the blocking probability of voice users. The voice user blocking probability of Algorithm 1 is worse than that of on-the-spot offloading and increases with $\lambda_{d}$. The blocking probability performance of Algorithm 2 is similar to that of the on-the-spot offloading. Since usually the voice users are blocked in the states where the only feasible action is blocking (say state $(C,0,0)$), the decision epochs where voice users are blocked are almost same as that of the on-the-spot offloading.

VI-C2 Total System Throughput Performance

In Fig. 3, total system throughputs for different algorithms are plotted as a function of $\lambda_{d}$.

In on-the-spot offloading, with an increase in $\lambda_{d}$, the number of WiFi data users increases, and this increases the total system throughput. However, for high $\lambda_{d}$, the effect of contention among data users reduces the rate of increment of the total system throughput. In Algorithm 1, as $\lambda_{d}$ increases, more number of data users are served using LTE. Since the throughput contribution of data users is more than voice users, the blocking probability of voice users increases with $\lambda_{d}$. Thus, the fraction of voice users in the system reduces, effectively causing more improvement in the total system throughput. When $\lambda_{d}=0.1$, the improvement in system metric is $25.22\%$, whereas for $\lambda_{d}=0.6$, the system metric almost doubles. In Fig. 3, the total system throughput values for Algorithm 2 are smaller than the corresponding values for Algorithm 1. The reduction in blocking probability of voice users comes at a price of the reduction in the total system throughput. Still, due to optimal association and load balancing decisions, Algorithm 2 reduces the effect of contention among data users in WiFi and hence performs better than on-the-spot offloading algorithm [11]. For example, with $\lambda_{d}=0.1$, the improvement in system metric is about $22.96\%$ and with $\lambda_{v}=0.6$, it becomes almost $93\%$.

VII Conclusion

In this paper, we have formulated the optimal association problem in an LTE-WiFi HetNet as an MDP problem with an objective of maximizing the total system throughput. Constrained MDP formulation has also been presented, where maximizing the total system throughput is subject to a constraint on the blocking probability of voice users. Threshold structures on the association of voice and data users have been derived. Based on the structure of the optimal policies, we have proposed two algorithms for the association and offloading of voice/data users in an LTE-WiFi HetNet. Simulation results demonstrate that although the voice user blocking probability performance of Algorithm 1 is worse than that of on-the-spot offloading [11], Algorithm 2 performs as good as on-the-spot offloading. Moreover, the proposed algorithms perform better than on-the-spot offloading algorithm in improving the total system throughput. In future, the considered framework can be extended to consider the channel state between LTE BS/WiFi AP and users such that channel-aware association and offloading decisions can be taken.

Appendix A Proof of Lemma 1

Since the decisions of association and offloading are involved during the arrival and departure of users, proving this lemma is equivalent to proving the following statements.

• (a)

$A_{3}$ (Accept in WiFi) is optimal when there are less than $k_{th}$ data users in the system, and a data user arrives.

• (b)

$A_{1}$ (Do nothing) is optimal when there are less than or equal to $k_{th}$ data users in the system, and a voice user from LTE departs.

• (c)

$A_{1}$ (Do nothing) is optimal when there are less than or equal to $k_{th}$ data users in the system, and a data user from WiFi departs.

We prove the above statements by sample path argument. Suppose the system starts at time $t=0$.

Proof of (a): We consider the scenario when the system is in the state $S_{1}=(i,0,0)$, when a data user arrival occurs (after a time $t_{1}$, say). Assume that the optimal policy $\pi^{*}$ does not associate this incoming data user with WiFi. Therefore, the optimal action must be $A_{2}$ (Accept in LTE). As the optimal policy is $\pi^{*}$, we have $V^{\pi^{*}}(s)\geq V^{\hat{\pi}}(s)$, $\forall\hat{\pi}\in\prod$ and $\forall s\in\mathcal{S}$, where $\prod$ is the set of all policies. Let us consider another policy $\pi$ (non-stationary in general) which takes $A_{3}$ in state $S_{1}=(i,0,0)$. As illustrated in Fig. 4, let us assume that starting from the state $S_{1}$ and following the policy $\pi^{*}$ and $\pi$, the system reaches the state $S_{2}=(i,1,0)$ and $S_{3}=(i,0,1)$, respectively. The inter-arrival times and service times are same for both the sample paths as we have considered a Markovian system. Assume that from the state $S_{2}$, based on the next event (after a time $t_{2}$) and the chosen action, the system makes a transition to the state $S_{4}$ according to the policy $\pi^{*}$. Before reaching the state $S_{2}$, the sample path followed by the policy $\pi^{*}$ has one less WiFi data user and one more LTE data user than that of the policy $\pi$ before it reaches the state $S_{3}$. Suppose, the policy $\pi$ is such that in state $S_{3}$, it takes the same action as that of policy $\pi^{*}$ and additionally offloads one data user from WiFi to LTE. Evidently, sample path followed by both the policies end up in the same state $S_{4}$. We construct $\pi$ in such a manner that from the state $S_{4}$ onwards, both the policies take up same actions and follow the same sample path. Therefore, the difference of value functions of the state $S_{1}$ under the policy $\pi^{*}$ and $\pi$ is

[TABLE]

Since $R_{L,D}<\tilde{R}_{W,D}(k),\forall k<k_{th}$ and $\tilde{R}_{W,D}(1)=R_{W,D}(1)$, we have, $V^{\pi^{*}}(S_{1})<V^{\pi}(S_{1})$. Clearly, this contradicts the original claim that $\pi^{*}$ is an optimal policy. Since the Markov chains induced by different policies are recurrent, the state $(i,0,0)$ is visited infinitely often and each time choice of $A_{3}$ upon a data user arrival provides more reward than action $A_{2}$. Therefore, when there is no data user in the system, and one data user arrives, $A_{3}$ is optimal. In a similar manner, it can be proved that $A_{3}$ is optimal when a data user arrives and the system is in state $(i,0,k)$, where $k<k_{th}$.

Proof of (b) and (c):These can be proved using a similar sample path argument.

Appendix B Proof of Lemma 2

Similar to Lemma 1, proving this lemma is equivalent to proving the following statements.

• (a)

$A_{2}$ (Accept in LTE) is optimal when there are more than or equal to $k_{th}$ data users in the system, and one data user arrives.

• (b)

$A_{1}$ (Do nothing) is optimal when there are more than $k_{th}$ data users in the system, and a voice/data user from LTE departs.

• (c)

$A_{5}$ (Data offload to a RAT from where a user has departed) is optimal when there are more than $k_{th}$ data users in the system, and a data user from WiFi departs.

Proof of (a): From Lemma 1, we have, $(j+k)\leq k_{th}\implies j=0$. We consider the scenario when the system is in the state $(i,0,k_{th})$, when a data user arrival occurs. Assume that the optimal policy $\pi^{*}$ does not associate this incoming data user with LTE. Consequently, the optimal action must be $A_{3}$. As the optimal policy is $\pi^{*}$, we have $V^{\pi^{*}}(s)\geq V^{\hat{\pi}}(s)$ $\forall\hat{\pi}\in\prod$ in every state $s$. Let us consider another policy $\pi$ which chooses $A_{2}$ in state $(i,0,k_{th})$. As illustrated in Fig. 4, starting from the state $(i,0,{k_{th}})$ and following the policy $\pi^{*}$ and $\pi$, the system reaches the states $S_{2}$ and $S_{3}$, respectively. From the state $S_{2}$, based on an event, the system reaches the state $S_{4}$. Suppose, in the state $S_{3}$, the action followed by policy $\pi$ is such that it chooses the same action as that of policy $\pi^{*}$ and additionally offloads one data user from LTE to WiFi. Clearly, path followed by both the policies end up in the same state $S_{4}$. We construct $\pi$ in such a way that from the state $S_{4}$ onwards, both of them follow the same path. Similar to the previous lemma, the difference of value functions under the policy $\pi^{*}$ and $\pi$ is

[TABLE]

Since $R_{L,D}\geq\tilde{R}_{W,D}(k),\forall k\geq k_{th}$, we have, $V^{\pi^{*}}(S_{1})<V^{\pi}(S_{1})$. Clearly, this contradicts the original claim that $\pi^{*}$ is an optimal policy. Thus, $A_{2}$ is optimal when there are $k_{th}$ data users in WiFi, and one data user arrives. The same result can be extended for the case when there are $k_{th}$ data users in WiFi, more than or equal to one data user in LTE, and one data user arrives.

Statements (b) and (c) can be proved in a similar way.

Appendix C Proof of Lemma 5

To prove this lemma, we consider two cases, $(1)$ $k\geq k_{th}$ and $(2)$ $k<k_{th}$. We prove the required for the first case. Proof of the second case follows in a similar manner. From Lemma 4, we know that for $k\geq k_{th}$, $A_{2}$ is better than $A_{4}$. Thus, for $k\geq k_{th}$, the choice is between $A_{1}$ and $A_{2}$. To prove this lemma, we first prove that the value function $V(i,j,k)$ is concave in $i$. In Lemma 1 and 2, we have already derived the structure of the optimal policy for data user arrival and departure of voice and data users. Now, for $k\geq k_{th}$, with the aid of this, the optimality equation is as follows.

[TABLE]

Let the components in Equation (6) be denoted by $V^{1}(i,j,k),V^{2}(i,j,k),V^{3}(i,j,k),V^{4}(i,j,k),V^{5}(i,j,k)$ and $V^{6}(i,j,k)$, respectively. We prove the concavity of $V(i,j,k)$ component-wise. Start the VIA with $V_{0}(i,j,k)=0$. Hence, $V_{0}(i,j,k)$ is concave in $i$. Let us assume that $V_{1,n}(i,j,k)=\max\{f(i,j,k)-\beta+V_{n-1}(i,j,k),f(i+1,j,k)+V_{n-1}(i+1,j,k)\}$. Equivalently, $V_{1,n}(i,j,k)=\max\{-\beta+V_{n-1}(i,j,k),R_{L,V}+V_{n-1}(i+1,j,k)\}$. Let us define the function $V_{1,n}(i,j,k,a)$ as follows.

[TABLE]

By definition,

[TABLE]

Thus, we have,

[TABLE]

Let us define $D_{i}V(i,j,k,a)=V(i+1,j,k,a)-V(i,j,k,a)$.

[TABLE]

[TABLE]

Since $V_{n-1}(i,j,k)$ is concave in $i$, $V_{1,n}(i,j,k,a)$ is concave in $i$.

Now, we need to prove that $V_{1,n}(i,j,k)$ is concave in $i$. In other words, we need to prove that $V_{1,n}(i+2,j,k)+V_{1,n}(i,j,k)\leq 2V_{1,n}(i+1,j,k)$. Let us assume that $a_{1}\in\{A_{1},A_{2}\}$ and $a_{2}\in\{A_{1},A_{2}\}$ are the maximizing actions in states $(i+2,j,k)$ and $(i,j,k)$, respectively. Therefore,

[TABLE]

Let us take $X=D_{i}V_{1,n}(i,j,k,a_{2})-D_{i}V_{1,n}(i+1,j,k,a_{1})$. To prove that $V_{1,n}(i,j,k)$ is concave in $i$, we need to prove that $X\geq 0$. There are four cases as described below.

$\text{Case }1:a_{1}=a_{2}=A_{1},$

[TABLE]

$\text{Case }2:a_{1}=A_{1},a_{2}=A_{2},$

[TABLE]

$\text{Case }3:a_{2}=a_{2}=A_{2},$

[TABLE]

$\text{Case }4:a_{1}=A_{2},a_{2}=A_{1},$

[TABLE]

Thus, it is proved that $V_{1,n}(i,j,k)$ is concave in $i$. Since this holds for every $n$ and every $\beta$, $V^{1}(i,j,k)$ is concave in $i$.

Similarly, in the case of the second component, let $V_{2,n}(i,j,k)=f(i+1,j,k)+V_{n-1}(i,j+1,k)$.

Thus, $D_{ii}V_{2,n}(i,j,k)=D_{ii}V_{n-1}(i,j+1,k).$ Therefore, $V_{2,n}(i,j,k)$ is concave in $i$. Similarly, other components also can be proved to be concave in $i$. Therefore, $V(i,j,k)$ is concave in $i$.

Let us define $x(i,j,k)=-\beta-R_{L,V}$. In order to prove this lemma, we know that if state $(i,j,k)$ is blocking, then $V(i+1,j,k)-V(i,j,k)\leq x(i,j,k).$ Due to concavity of $V(i,j,k)$, $V(i+2,j,k)-V(i+1,j,k)\leq V(i+1,j,k)-V(i,j,k)$. Now, $x(i,j,k)=x(i+1,j,k)$. As a consequence, $V(i+2,j,k)-V(i+1,j,k)\leq x(i+1,j,k).$ Thus, it is proved that if state $(i,j,k)$ is blocking, then the state $(i+1,j,k)$ is also blocking.

To prove that if state $(i,j,k)$ is blocking, then the state $(i,j+1,k)$ is also blocking, we first need to prove that the value function is submodular in $(i,j)$. In other words, we need to prove that $V_{n}(i+1,j,k)+V_{n}(i,j+1,k)\geq V_{n}(i,j,k)+V_{n}(i+1,j+1,k)$. Similar to the previous proof, we prove the above statement component-wise. Let us assume that $a_{1}$ and $a_{2}$ are the maximizing actions in states $(i,j,k)$ and $(i+1,j+1,k)$, respectively. Start the VIA with $V_{0}(i,j,k)=0$. Therefore, $V_{0}(i,j,k)$ is submodular in $(i,j)$. In other words, $D_{ij}V_{0}(i,j,k)\leq 0$. We have,

[TABLE]

Now, we consider four possible cases.

$\text{Case }1:a_{1}=a_{2}=A_{1},$

[TABLE]

$\text{Case }2:a_{1}=a_{2}=A_{2},$

[TABLE]

$\text{Case }3:a_{1}=A_{1},a_{2}=A_{2},$

[TABLE]

$\text{Case }4:a_{1}=A_{2},a_{2}=A_{1},$

[TABLE]

Thus, it is proved that $V_{1,n}(i,j,k)$ is submodular in $(i,j)$.

Similarly, in the case of the second component, we have, $V_{2,n}(i,j,k)=f(i,j+1,k)+V_{n}(i,j+1,k)$. Therefore, we have, $D_{ij}V_{2,n}(i,j,k)=D_{ij}V_{n}(i,j+1,k)\leq 0$. Similarly, other components also can be proved to be submodular in $(i,j)$. Therefore, the value function is submodular in $(i,j)$.

Now, if state $(i,j,k)$ is blocking then we have, $V(i+1,j,k)-V(i,j,k)\leq x(i,j,k).$ Again, we have, $x(i,j,k)=x(i,j+1,k)$. Due to submodularity, we have $V(i+1,j+1,k)-V(i,j+1,k)\leq V(i+1,j,k)-V(i,j,k)\leq x(i,j,k)=x(i,j+1,k)$. Thus, in the case of voice arrival, if $A_{1}$ is optimal in state $(i,j,k)$, then in state $(i,j+1,k)$ also $A_{1}$ is optimal.

Proof of $(ii)$ follows directly from the proof of part $(i)$.

Appendix D Proof of Lemma 6

To prove this lemma, we consider two cases, $(1)$ $k\geq k_{th}$ and $(2)$ $k<k_{th}$. We demonstrate the proof of the lemma for the first case. Proof of the second case follows in similar manner. To prove this lemma, we first need to prove that for $(i+j)=C$, the difference of value functions $V(i+1,j-1,k+1)-V(i,j,k)$ is decreasing in $i$. For $(i+j)=C$ and $k\geq k_{th}$, the optimality equation can be described as

[TABLE]

Let us assume that $V_{1,n}(i,j,k,a)=\max\{f(i,j,k)-\beta+V_{n-1}(i,j,k),f(i+1,j-1,k+1)+V_{n-1}(i+1,j-1,k+1)\}$.

Equivalently, $V_{1,n}(i,j,k,a)=\max\{-\beta+V_{n-1}(i,j,k),\\ R_{L,V}-R_{L,D}+\tilde{R}_{W,D}(k)+V_{n-1}(i+1,j-1,k+1)\}$.

In other words,

[TABLE]

We prove the above claim component-wise. Start the VIA with $V_{0}(i,j,k)=0$. Therefore, $V_{0}(i+1,j-1,k+1)-V_{0}(i,j,k)$ is decreasing in $i$. We also have, $E_{i}F_{i}V_{0}(i,j,k)\leq 0$. Now, Let us define $E_{i}V(i,j,k,a)=V(i+1,j-1,k+1,a)-V(i,j,k,a)$ and $F_{i}V(i,j,k,a)=V(i+1,j-1,k,a)-V(i,j,k,a)$.

[TABLE]

Therefore, $E_{i}F_{i}V_{1,n}(i,j,k,a)\leq 0$.

[TABLE]

Therefore, $V_{1,n}(i+1,j-1,k+1,a)-V_{1,n}(i,j,k,a)$ is decreasing in $i$.

Now, we need to prove that $V_{1,n}(i+1,j-1,k+1)-V_{1,n}(i,j,k)$ is decreasing in $i$. In other words, we need to prove that $V_{1,n}(i+2,j-2,k+1)+V_{1,n}(i,j,k)\leq V_{1,n}(i+1,j-1,k+1)+V_{1,n}(i+1,j-1,k)$. Let us assume that $a_{1}\in\{A_{1},A_{4}\}$ and $a_{2}\in\{A_{1},A_{4}\}$ are the maximizing actions in states $(i+2,j-2,k+1)$ and $(i,j,k)$, respectively. Therefore,

[TABLE]

Let us take $Y=E_{i}V_{1,n}(i,j,k,a_{2})-E_{i}V_{1,n}(i+1,j-1,k,a_{1})$. To prove that $V_{1,n}(i+1,j-1,k+1)-V_{1,n}(i,j,k)$ is decreasing in $i$, we need to prove that $Y\geq 0$. There are four cases as described below.

$\text{Case }1:a_{1}=a_{2}=A_{1},$

[TABLE]

$\text{Case }2:a_{1}=A_{4},a_{2}=A_{1},$

[TABLE]

$\text{Case }3:a_{2}=a_{2}=A_{4},$

[TABLE]

$\text{Case }4:a_{1}=A_{1},a_{2}=A_{4},$

[TABLE]

Thus, it is proved that $V_{1,n}(i+1,j-1,k+1)-V_{1,n}(i,j,k)$ is decreasing in $i$. Since this holds for every $n$ and every value of $\beta$, $V^{1}(i+1,j-1,k+1)-V^{1}(i,j,k)$ is decreasing in $i$.

Let $V_{2,n}(i,j,k)=f(i,j,k+1)+V_{n-1}(i,j,k+1)$. Thus, $E_{ii}V_{2,n}(i,j,k)=E_{ii}V_{n-1}(i,j,k+1).$ Therefore, $V_{2,n}(i+1,j-1,k+1)-V_{2,n}(i,j,k)$ is decreasing in $i$. Similarly, other components can be proved to be decreasing in $i$. Therefore, $V(i+1,j-1,k+1)-V(i,j,k)$ is decreasing in $i$.

Similar to Lemma 5, using this property it can be proved that if the optimal action for voice user arrival in state $(i,j,k)$ is blocking, then the optimal action in state $(i+1,j-1,k)$ is also blocking.

Proof of $(ii)$ follows directly from the proof of part $(i)$.

Acknowledgment

This work is funded by the Department of Electronics and Information Technology (DeitY), Government of India.