Decentralized Computation Offloading for Multi-User Mobile Edge Computing: A Deep Reinforcement Learning Approach

TL;DR

This paper proposes a deep reinforcement learning approach using DDPG for decentralized computation offloading in multi-user MEC systems, effectively reducing power consumption and delay through adaptive power allocation.

Contribution

It introduces a novel DDPG-based decentralized offloading strategy that independently learns efficient policies at each user, outperforming existing methods.

Findings

DDPG-based strategy reduces computation cost compared to DQN and greedy methods.

Efficient policies are learned independently at each user.

Power-delay tradeoff is analyzed for the proposed strategies.

Abstract

Mobile edge computing (MEC) emerges recently as a promising solution to relieve resource-limited mobile devices from computation-intensive tasks, which enables devices to offload workloads to nearby MEC servers and improve the quality of computation experience. Nevertheless, by considering a MEC system consisting of multiple mobile users with stochastic task arrivals and wireless channels in this paper, the design of computation offloading policies is challenging to minimize the long-term average computation cost in terms of power consumption and buffering delay. A deep reinforcement learning (DRL) based decentralized dynamic computation offloading strategy is investigated to build a scalable MEC system with limited feedback. Specifically, a continuous action space-based DRL approach named deep deterministic policy gradient (DDPG) is adopted to learn efficient computation offloading policies independently at each mobile user. Thus, powers of both local execution and task offloading can be adaptively allocated by the learned policies from each user's local observation of the MEC system. Numerical results are illustrated to demonstrate that efficient policies can be learned at each user, and performance of the proposed DDPG based decentralized strategy outperforms the conventional deep Q-network (DQN) based discrete power control strategy and some other greedy strategies with reduced computation cost. Besides, the power-delay tradeoff is also analyzed for both the DDPG based and DQN based strategies.

Tables

Table 1. TABLE I: Summary of Notations

Notation	Description
$ℳ$	The set of mobile users
$𝒯$	Index set of the time slots
${𝒉}_m\left(t\right)$	Channel vector between user $m$ and the BS at slot $t$
$𝒚\left(t\right)$	Received signal of the BS at slot $t$
${\rho }_m$	Normalized temporal channel correlation coefficient of user $m$
$𝑯\left(t\right)$	Channel matrix from all the users to the BS at slot $t$
${𝒈}_m\left(t\right)$	ZF detection vector for user $m$ at slot $t$
${\gamma }_m\left(t\right)$	The receiving SINR of user $m$ at slot $t$
${\lambda }_m$	Task arrival rate of user $m$
$B_m\left(t\right)$	Queue length of user $m$’s task buffer at slot $t$
$a_m\left(t\right)$	Number of task arrivals of user $m$ at slot $t$
$f_m\left(t\right)$	CPU frequency scheduled for local execution of user $m$ at slot $t$
$L_m$ $\left(F_m\right)$	CPU cycles required per one task bit (allowable CPU-cycle frequency) at user $m$
$p_{o,m}\left(t\right)$	Transmission power of user $m$ for computation offloading at slot $t$
$d_{o,m}\left(t\right)$	Data transmitted by user $m$ for computation offloading at slot $t$
$p_{l,m}\left(t\right)$	Power consumption of user $m$ for local execution at slot $t$
$d_{l,m}\left(t\right)$	Data processed by user $m$ via local execution at slot $t$
$P_{o,m}$ $\left(P_{l,m}\right)$	Maximum transmission power (local execution power) of user $m$
${\tau }_0$	Length of one time slot

Table 2. TABLE II: Comparison of testing results with w m = 0.5 subscript 𝑤 𝑚 0.5 w_{m}=0.5 for all mobile users.

	Average Reward			Average Power			Average Delay
	User 1	User 2	User 3	User 1	User 2	User 3	User 1	User 2	User 3
DDPG	-1.770	-5.670	-12.782	0.205	0.774	1.939	1.489	3.600	6.174
DQN	-2.174	-7.657	-14.688	0.292	1.156	2.320	1.428	3.753	6.176
GD-Offload	-2.514	-7.597	-18.690	0.402	1.309	3.143	1.007	2.103	5.951
GD-Local	-1.633	-9.504	-20.071	0.216	1.678	3.407	1.106	2.228	6.072

Table 3. TABLE III: Comparison of testing results with w m = 0.8 subscript 𝑤 𝑚 0.8 w_{m}=0.8 for all mobile users.

	Average Reward			Average Power			Average Delay
	User 1	User 2	User 3	User 1	User 2	User 3	User 1	User 2	User 3
DDPG	-1.919	-6.366	-16.164	0.162	0.602	1.674	3.114	7.752	13.861
DQN	-2.780	-8.915	-18.675	0.284	0.915	1.954	2.539	7.973	15.216
GD-Offload	-3.417	-10.893	-26.334	0.402	1.309	3.143	1.007	2.103	5.951
GD-Local	-1.949	-13.870	-28.470	0.216	1.678	3.407	1.106	2.228	6.072

Equations

$$R_t=\sum _{i=t}^T{\gamma }^{i-t}r(s_i,a_i),$$

$$V^{\pi }(s_t,a_t)={\mathbb{E}}_{s_{i>t}\sim E,a_{i\ge t}\sim \pi }\left[R_t|s_t\right],$$

$$Q^{\pi }(s_t,a_t)={\mathbb{E}}_{s_{i>t}\sim E,a_{i>t}\sim \pi }\left[R_t|s_t,a_t\right].$$

$$V^{\pi }(s_t)$$

$$Q^{\pi }(s_t,a_t)$$

$$V^{\ast }(s_t)=\underset{a_t\in \mathcal{A}}{\max }{\mathbb{E}}_{s_{t+1}\sim E}\left[r(s_t,a_t)+\gamma V^{\ast }(s_{t+1})\right].$$

$$V_{k+1}(s)=\underset{a\in \mathcal{A}}{\max }\sum _{s^{\prime }}p(s^{\prime }|s,a)\left[r(s,a)+\gamma V_k(s^{\prime })\right],$$

$${\pi }^{\ast }(s)=\arg \underset{a\in \mathcal{A}}{\max }{Q}^{\ast }(s,a).$$

$$Q^{\ast }(s_t,a_t)={\mathbb{E}}_{s_{t+1}\sim E}\left[r(s_t,a_t)+\gamma \underset{a_{t+1}}{\max }{Q}^{\ast }(s_{t+1},a_{t+1})\right],$$

$$Q(s_t,a_t)\leftarrow Q(s_t,a_t)+\alpha \left[r(s_t,a_t)+\gamma \underset{a_{t+1}}{\max }Q(s_{t+1},a_{t+1})-Q(s_t,a_t)\right],$$

$$L(\theta )={\mathbb{E}}_{(s,a,r,s^{\prime })\sim U(\mathcal{B})}\left[{\left(r+\gamma \underset{a\in \mathcal{A}}{\max }Q(s^{\prime },a|{\theta }^{\prime })-Q(s,a|\theta )\right)}^2\right],$$

$${\nabla }_{{\theta }^{\mu }}J\approx {\mathbb{E}}_{(s,a,r,s^{\prime })\sim U(\mathcal{B})}\left[{\nabla }_{a}Q(s,a|{\theta }^Q){\nabla }_{{\theta }^{\mu }}\mu (s|{\theta }^{\mu })\right],$$

$${\mu }^{\prime }(s)=\mu (s|{\theta }^{\mu })+\Delta \mu ,$$

$$\mathbf{y}(t)=\sum _{m=1}^M{\mathbf{h}}_m(t)\sqrt{p_{o,m}(t)}{s}_m(t)+\mathbf{n}(t),$$

$${\mathbf{h}}_m(t)={\rho }_m{\mathbf{h}}_m(t-1)+\sqrt{1-{\rho }_m^2}\mathbf{e}(t),$$

$${\gamma }_m(t)=\frac{p_{o,m}(t)}{{\sigma }_R^2\Vert {\mathbf{g}}_m(t){\Vert }^2}=\frac{p_{o,m}(t)}{{\sigma }_R^2{\left[{\left({\mathbf{H}}^H(t)\mathbf{H}(t)\right)}^{-1}\right]}_{mm}},$$

$$B_m(t+1)={\left[B_m(t)-\left(d_{l,m}(t)+d_{o,m}(t)\right)\right]}^{+}+a_m(t),\forall t\in \mathcal{T},$$

$$f_m(t)=\sqrt[3]{p_{l,m}(t)/\kappa },$$

$$d_{l,m}(t)={\tau }_0{f}_m(t)L_m^{-1}.$$

$$d_{o,m}(t)={\tau }_{0}W{\log }_2\left(1+{\gamma }_m(t)\right),$$

$$s_{m,t}=\left[B_m(t),{\varphi }_m(t-1),{\mathbf{h}}_m(t)\right],$$

$${\varphi }_m(t)=\frac{{\gamma }_m(t){\sigma }_R^2}{p_{o,m}(t)\Vert {\mathbf{h}}_m(t){\Vert }^2}=\frac{1}{\Vert {\mathbf{h}}_m(t){\Vert }^2{\left[{\left({\mathbf{H}}^H(t)\mathbf{H}(t)\right)}^{-1}\right]}_{mm}}.$$

$$a_{m,t}=\left[p_{l,m}(t),p_{o,m}(t)\right].$$

$$r_{m,t}=-w_{m,1}\cdot \left(p_{l,m}(t)+p_{o,m}(t)\right)-w_{m,2}\cdot B_m(t),$$

$$V^{{\mu }_m}(s_{m,1})=\mathbb{E}\left[\sum _{t=1}^{\infty }{\gamma }^{t-1}{r}_{m,t}|s_{m,1}\right],$$

$${\bar{C}}_m(s_{m,t})=\mathbb{E}\left[\underset{T\to \infty }{\lim }\frac{1}{T}\sum _{i=t}^T{w}_{m,1}\left(p_{l,m}(i)+p_{o,m}(i)\right)+w_{m,2}{B}_m(i)|s_{m,t}\right],$$

$$L=\frac{1}{I}\sum _{i=1}^I{\left(r_i+\underset{a\in \mathcal{A}}{\max }Q(s_i^{\prime },a|{\theta }_m^{Q^{\prime }})-Q(s_i,a_i|{\theta }_m^Q)\right)}^2;$$

$${\nabla }_{{\theta }_m^{\mu }}J\approx \frac{1}{I}\sum _{i=1}^I{\nabla }_{a}Q(s_i,a|{\theta }_m^Q){|}_{a=a_i}{\nabla }_{{\theta }_m^{\mu }}\mu (s_i|{\theta }_m^{\mu });$$

$$r_{m,t}=-10w_m\cdot \left(p_{l,m}(t)+p_{o,m}(t)\right)-(1-w_m)\cdot B_m(t),$$

Full text

I Introduction

As the popularity of smart mobile devices in the coming 5G era, mobile applications, especially for computation-intensive tasks such as online 3D gaming, face recognition and location-based augmented or virtual reality (AR/VR), have been greatly affected by the limited on-device computation capability [1]. Meanwhile, for the large number of low-power and resource-constrained wireless terminals serving in the emerging Internet of Things (IoT) [2] and Intelligent Transport Systems (ITS) [3], a huge amount of sensory data also needs to be pre-processed and analyzed. As a result, to meet the quality of experience (QoE) of these mobile applications, the technology of mobile edge computing (MEC) [4] has been proposed as a promising solution to bridge the gap between the limited resources on mobile devices and the ever-increasing demand of computation requested by mobile applications.

Instead of the remote public clouds in conventional cloud computing systems such as Amazon Web Services and Microsoft Azure, MEC enhances the radio access networks (RANs), which is in close proximity to mobile users, with computing capability [5, 6, 7]. It enables mobile devices to offload computation workloads to the MEC server associated with a base station (BS), and thus improves the QoE of mobile applications with considerably reduced latency and power consumption. Many researcher has been attracted from both the industry [8] and academia [9, 10, 11]. Nevertheless, computation offloading highly depends on the efficiency of wireless data transmission, which requires MEC systems to manage radio resources along with computation resources and complete computation tasks efficiently.

In order to achieve higher energy efficiency or better computation experience, computation offloading strategies for MEC have been widely investigated in the literature recently. For short-term optimization over quasi-static channels, some algorithms have been studied in [12, 13, 14, 15, 16, 17, 18]. In [12], optimal joint offloading selection and radio resource allocation for mobile task offloading was studied to minimize the overall execution time. For decentralized algorithms with reduced overhead, a game-theoretic computation offloading scheme was constructed in [13]. Moreover, with the dynamic voltage and frequency (DVFS) techniques, CPU-cycle frequency was flexibly controlled with other features in [14, 15], where the system cost, defined as weighted sum of energy consumption and execution time, has been reduced. Besides, energy-latency tradeoff has been discussed in [16] with jointly optimized communication and computation resource allocation under the limited energy and sensitive latency. Also, it has been shown the performance of MEC can be further improved with adopting some other emerging technologies such as wireless power transfer [17] and non-orthogonal multiple access (NOMA) [18].

To cope with stochastic task arrivals and time-varying wireless channels, strategies for dynamic joint control of radio and computation resources in MEC systems become even challenging [19, 20, 21, 22, 23, 24, 25, 26, 27, 28]. In [19], a threshold-based dynamic computation offloading policy was proposed to minimize energy consumption under stochastic wireless channels. For low-complexity online algorithms, Lyapunov optimization has been widely adopted. In [20], dynamic policies for offloading decision, clock speed and network interface control were considered to minimize energy consumption with given delay constraints. Joint optimization of multiple-input multiple-output (MIMO) beamforming and computational resource allocation for a multi-cell MEC system is designed in [21]. Additionally, an energy harvesting enabled green MEC system is studied in [22], where the delay cost addressing both the execution delay and task failure is minimized. For multiple user scenarios, power-delay tradeoff [23], network utility maximization balancing throughput and fairness with reduced feedback [24], and stochastic admission control and scheduling for multi-user multi-task computation offloading [25] were discussed, respectively. On the other hand, Markov decision process (MDP) can be also applied to the analysis and design of dynamic control of MEC systems [26, 27]. Furthermore, it was shown in [28] and [29] that an optimal dynamic computation offloading policy can be learned by the emerging reinforcement learning (RL) based algorithm without any prior knowledge of the MEC system.

Conventional RL algorithms cannot scale well as the number of agents increases, since the explosion of state space will make traditional tabular methods infeasible [30]. Nevertheless, by exploiting deep neural networks (DNNs) for function approximation, deep reinforcement learning (DRL) has been demonstrated to efficiently approximate Q-values of RL [31]. There have been some attempts to adopt DRL in the design of online resource allocation and scheduling in wireless networks [32, 33, 34], especially for some recent works targeting computation offloading in MEC [35, 36, 37, 38]. Specifically, in [38], system sum cost combining execution delay and energy consumption of a multi-user MEC system is minimized by optimal offloading decision and computational resource allocation. Similarly, the authors in [36] considered an online offloading algorithm to maximize the weighted sum computation rate in a wireless powered MEC system. In [37], a DRL based computation offloading strategy of an IoT device is learned to choose one MEC server to offload and determine the offloading rate. Besides, double deep Q-network (DQN) based strategic computation offloading algorithm was proposed in [35], where an mobile device learned the optimal task offloading and energy allocation to maximize the long-term utility based on the task queue state, the energy queue state as well as the channel qualities. In the existing works, there have been only strategies focusing on centralized DRL based algorithms for optimal computation offloading in MEC systems, and the design of decentralized DRL based algorithms for dynamic task offloading control of a multi-user MEC system still remains unknown.

In this paper, we consider a general MEC system consisting of one base station (BS) with one attached MEC server and multiple mobile users, where tasks arrive stochastically and channel condition is time-varying for each user. Without any prior knowledge of network statistics of the MEC system, a dynamic computation offloading policy will be learned independently at each mobile user based on local observations of the MEC system. Moreover, different from other DRL based policies in existing works making decisions in discrete action spaces, we adopt a continuous action space based algorithm named deep deterministic policy gradient (DDPG) to derive better power control of local execution and task offloading. Specifically, major contributions of this paper can be summarized as follows:

• •

A multi-user MIMO based MEC system is considered, where each mobile user with stochastic task arrivals and time-varying wireless channels attempts to independently learn dynamic computation offloading policies from scratch to minimize long-term average computation cost in terms of power consumption and task buffering delay.

• •

By adopting DDPG, a DRL framework for decentralized dynamic computation offloading has been designed, which enables each mobile user to leverage only local observations of the MEC system to gradually learn efficient policies for dynamic power allocation of both local execution and computation offloading in a continuous domain.

• •

Numerical simulations are performed to illustrate performance of the policy learned from the DDPG based decentralized strategy and analyze the power-delay tradeoff for each user. Superiority of the continuous power control based DDPG over the discrete control based DQN and some other greedy strategies is also demonstrated.

The rest of this paper is organized as follows. In Section II, some preliminaries on DRL are introduced. Then, system model for the dynamic computation offloading of the MEC system is presented in Section III. Design of the decentralized DRL based dynamic computation offloading algorithm is proposed in Section IV. Numerical results will be illustrated in Section V. Finally, Section VI concludes this paper.

II Preliminaries on Deep Reinforcement Learning

In this section, we will firstly give an overview of MDP and RL [30], and then introduce some basics of the emerging DRL technology [31]. Finally, recent extension of DRL on continuous action space based algorithm, i.e., DDPG [39], is presentedX

II-A MDP

An MDP consists of an agent and an environment $E$, a set of possible states $\mathcal{S}$, a set of available actions $\mathcal{A}$, and a reward function $r:\mathcal{S}\times\mathcal{A}\rightarrow\mathcal{R}$, where the agent continually learns and makes decisions from the interaction with the environment in discrete time steps. In each time step $t$, the agent observes current state of the environment as $s_{t}\in\mathcal{S}$, and chooses and executes a action $a_{t}\in\mathcal{A}$ according to a policy $\pi$. After that, the agent will receive a scalar reward $r_{t}=r(s_{t},a_{t})\in\mathcal{R}\subseteq\mathbb{R}$ from the environment $E$ and find itself in the next state $s_{t+1}\in\mathcal{S}$ according to the transition probability of the environment $p(s_{t+1}|s_{t},a_{t})$. Thus, the dynamics of the environment $E$ is determined by the transition probability as response to the action taken by the agent in the current state, while the goal of the agent is to find the optimal policy that maximizes the long-term expected discounted reward it receives, i.e.,

[TABLE]

where $T\rightarrow\infty$ is the total number of time steps taken and $\gamma\in[0,1]$ is the discounting factor. It is worth noting that a policy $\pi$ is generally stochastic, which maps the current state to a probability distribution over the actions, i.e., $\pi:\mathcal{S}\rightarrow\mathcal{P}(\mathcal{A})$. Under the policy $\pi$, the expected discounted return starting from state $s_{t}$ is defined as the value function

[TABLE]

while the state-action function is the expected discounted return after taking an action $a_{t}$, i.e.,

[TABLE]

A fundamental property which is frequently used in MDP called Bellman equation represents the recursive relationship of the value function and the state-action function, respectively,

[TABLE]

Moreover, under the optimal policy $\pi^{*}$, the Bellman optimality equation for the value function can be written by

[TABLE]

Based on the assumption of perfect model of the environment in MDP, dynamic programming (DP) algorithms like value iteration can be app∫lied to obtain the optimal value function of any state $s\in\mathcal{S}$ under the optimal policy $\pi^{*}$, i.e.,

[TABLE]

where $k$ is the index for value iteration. Once the optimal value function $V^{*}(s)$ is obtained, the optimal state-value function can be derived by $Q^{*}(s,a)=\mathbb{E}_{s^{\prime}\sim E}\left[r(s,a)+\gamma V^{*}(s^{\prime})\right]$. Then, it can be found that the optimal policy $\pi^{*}$ chooses the optimal action greedily in state $s$ as follows 111Note that this gives an special case for deterministic policies, which can be readily extended to stochastic policies. Specifically, the value iteration in (6) still holds for stochastic policies. If there are ties for different actions that maximized the value function, each maximizing action can be given a portion of probability to be selected, while other actions is selected with zero probability.,

[TABLE]

II-B RL

Unlink MDP, RL algorithms attempt to derive optimal policies without an explicit model of the environment’s dynamics. In this case, the underlying transmission probability $p(s_{t+1}|s_{t},a_{t})$ is unknown and even non-stationary. Thus, the RL agent will learn from actual interactions with the environment and adapting its behavior upon experiencing the outcome of its actions, so as to maximize the expected discounted rewards. To this end, as a combination of Monte Carlo methods and DP, temporal-difference (TD) method arises to learn directly from raw experience. Note that similar to (6), the Bellman optimality equation for the state-action function is

[TABLE]

from which we can update the state-action function using the agent’s experience tuple $(s_{t},a_{t},r_{t},s_{t+1})$ and other learned estimates at each time step $t$ as follows,

[TABLE]

where $\alpha$ is the learning rate and the value $\delta_{t}=r(s_{t},a_{t})+\gamma\max_{a_{t+1}}Q(s_{t+1},a_{t+1})-Q(s_{t},a_{t})$ is the TD error. The algorithm in (10) is the well-known Q-learning [40], from which the state-action function is also known as Q-value. It is worth noting that Q-learning is off-policy, since it directly approximates the optimal Q-value and the transitions experienced by the agent are independent of the policy being learned. Besides, it can be proved that Q-learning algorithm converges with probability one [30]. With the estimated optimal state-action function, the optimal policy $\pi^{*}$ can be easily obtained from (8).

II-C DRL

Thanks to the powerful function approximation properties of DNNs, DRL algorithms [31] can learn low-dimensional representations for RL problems, which addresses the curse of dimensionality efficiently. As illustrated in Fig. 1, the recent DQN technology [31] successfully takes advantage of a DNN parameterized by $\theta$ to approximate the Q-values $Q(s,a)$. In order to resolve the problem of instability of using function approximation in RL, an experience replay buffer $\mathcal{B}$ is employed, which stores the agent’s experiences $e_{t}=(s_{t},a_{t},r_{t},s_{t+1})$ at each time step $t$. Meanwhile, a mini-batch of samples $(s,a,r,s^{\prime})\sim U(\mathcal{B})$ will be drawn uniformly at random from $\mathcal{B}$, and the following loss function will be calculated:

[TABLE]

which can be used to update the network parameter by $\theta\leftarrow\theta-\alpha\cdot\nabla_{\theta}L(\theta)$ with a learning rate $\alpha$. Note that in order to further improve the stability of RL, the original DNN in DQN utilizes a target network $\theta^{\prime}$ to derive the TD error for the agent, which adopts a so-called soft update strategy that tracks the weights of the learned network by $\theta^{\prime}\leftarrow\tau\theta+(1-\tau)\theta^{\prime}$ with $\tau\ll 1$. Besides, the action taken by the agent at each time step $t$ is obtained by $a_{t}=\arg\max\limits_{a\in\mathcal{A}}Q(s_{t},a|\theta)$.

II-D DDPG

Although problems in high-dimensional state spaces has been successfully solved by DQN, only discrete and low-dimensional action spaces can be handled. To extend DRL algorithms to continuous action space, DDPG has been proposed in [39]. As shown in Fig. 2, an actor-critic approach is adopted by using two separate DNNs to approximate the Q-value network $Q(s,a|\theta^{Q})$, i.e., the critic function, and the policy network $\mu(s|\theta^{\mu})$, i.e., the actor function, respectively. 222Note that similar to DQN, both the original DNNs in DDPG also have its own target network parameterized by $\theta^{Q^{\prime}}$ and $\theta^{\mu^{\prime}}$, respectively, which use soft update strategy and slowly track the weights of the learned networks in the same way. Specifically, the critic $Q(s,a|\theta^{Q})$ is similar to DQN and can be updated following (11). On the other hand, the actor $\mu(s|\theta^{\mu})$ deterministically maps state $s$ to a specific continuous action. As derived in [41], the policy gradient of the actor can be calculated by chain rule,

[TABLE]

which is the gradient of the expected return from the start distribution $J$ with respect to the actor parameter $\Theta^{\mu}$, averaged over the sampled mini-batch $U(\mathcal{B})$. Thus, with (11) and (12) in hand, network parameters of the actor and the critic can be updated by $\theta^{Q}\leftarrow\theta^{Q}-\alpha_{Q}\cdot\nabla_{\theta^{Q}}L(\theta^{Q})$ and $\theta^{\mu}\leftarrow\theta^{\mu}-\alpha_{\mu}\cdot\nabla_{\theta^{\mu}}J$, respectively. Here, $\alpha_{Q}$ and $\alpha_{\mu}$ are the learning rates.

In order to improve the model with adequate exploration of the state space, one major challenge in RL is the tradeoff between exploration and exploitation [42], which is even more difficult for learning in continuous action spaces. As an off-policy algorithm, the exploration of DDPG can be treated independently from the learning process. Thus, the exploration policy $\mu^{\prime}$ can be constructed by adding noise $\Delta\mu$ sampled ∫from a random noise process to the actor, i.e.,

[TABLE]

where the random noise process needs to be elaborately selected. E.g., exploration noise sampled from a temporally correlated random process can better preserve momentum [39].

III Dynamic Computation Offloading for Mobile Edge Computing

As shown in Fig. 3, we consider a multi-user MEC system, which consists of a BS, an MEC server and a set of mobile users $\mathcal{M}=\{1,2,\ldots,M\}$. For each user $m\in\mathcal{M}$, it has computation-intensive tasks to be completed. Considering the limited computational resources on each mobile device, the MEC server is deployed in proximity to the BS by the telecom operator, which can improve user’s computation experience by enabling it to offload part of its computation need to the MEC server via the wireless link [6]. A discrete-time model is adopted for the MEC system, where the operating period is slotted with equal length $\tau_{0}$ and indexed by $\mathcal{T}=\{0,1,\ldots\}$. The channel condition and task arrival of each user varies for each slot $t\in\mathcal{T}$. Thus, aiming to balance the average energy consumption and task processing delay, each user needs to determine the ratio of local execution and computation offloading at each slot. Moreover, as the number of mobile users increases, decentralized task scheduling at each user is more favorable, which can reduce the system overhead between the users and the MEC server and improve the scalability of the MEC system. In the following parts, we will introduce the modeling of networking and computing in detail.

III-A Network Model

In the MEC system, we consider a 5G macro-cell or small-cell BS, which is equipped with $N$ antennas and manages the uplink transmissions of multiple single-antenna mobile users by employing the well-known linear detection algorithm zero-forcing (ZF), which is of low complexity but efficient, especially for multi-user MIMO with large antenna arrays [43]. For each time slot $t\in\mathcal{T}$, if the channel vector of each mobile user $m\in\mathcal{M}$ is represented by $\boldsymbol{h}_{m}(t)\in\mathbb{C}^{N\times 1}$, the received signal of the BS can be written as

[TABLE]

where $p_{o,m}(t)\in[0,P_{o,m}]$ is the transmission power of user $m$ to offload task data bits with $P_{o,m}$ being the maximum value, $s_{m}(t)$ is the complex data symbol with unit variance, and $\boldsymbol{n}(t)\sim\mathcal{CN}(\boldsymbol{0},\sigma_{R}^{2}\boldsymbol{I}_{N})$ is a vector of additive white Gaussian noise (AWGN) with variance $\sigma_{R}^{2}$. Note that $\boldsymbol{I}_{N}$ denotes an $N\times N$ identity matrix. In order to characterize the temporal correlation between time slots for each mobile user $m\in\mathcal{M}$, the following Gaussian Markov block fading autoregressive model [44] is adopted:

[TABLE]

where $\rho_{m}$ is the normalized channel correlation coefficient between slots $t$ and $t-1$, and the error vector $\boldsymbol{e}(t)$ is complex Gaussian and uncorrelated with $\boldsymbol{h}_{m}(t)$. Note that $\rho_{m}=J_{0}(2\pi f_{d,m}\tau_{0})$ according to Jake’s fading spectrum, where $f_{d,m}$ is the Doppler frequency of user $m$, $t_{0}$ is the slot length, and $J_{0}(\cdot)$ is the Bessel function of the first kind [45].

Denoting $\boldsymbol{H}(t)=[\boldsymbol{h}_{1}(t),\ldots,\boldsymbol{h}_{M}(t)]$ as the $N\times M$ channel matrix between the BS and the $M$ users, the linear ZF detector at the BS 333Here, we consider the number of antennas at the BS is larger than the number of users, i.e., $N>M$. According to [43], it is demonstrated that as $N$ increases, the SINR performance of ZF detection approaches that of the MMSE detection. can be written by the channel matrix’s pseudo inverse $\boldsymbol{H}^{\dagger}(t)=\left(\boldsymbol{H}^{H}(t)\boldsymbol{H}(t)\right)^{-1}\boldsymbol{H}^{H}(t)$. If the $m$-th row of $\boldsymbol{H}^{\dagger}(t)$ is represented by $\boldsymbol{g}^{H}_{m}(t)$, the received signal for user $m$ is $\boldsymbol{g}^{H}_{m}(t)\boldsymbol{y}(t)=\sqrt{p_{o,m}(t)}s_{m}(t)+\boldsymbol{g}^{H}_{m}(t)\boldsymbol{n}(t)$, since we have $\boldsymbol{g}^{H}_{i}(t)\boldsymbol{h}_{j}(t)=\delta_{ij}$ for ZF detection [43]. Here, $\delta_{ij}=1$ when $i=j$ and [math] otherwise. Thus, the corresponding signal-to-interference-plus-noise (SINR) can be derived by

[TABLE]

where $[\boldsymbol{A}]_{mn}$ is the $(m,n)$-th element of matrix $\boldsymbol{A}$. From (16), it can be verified that each user’s SINR will become worse as the number of users $M$ increases, which requires each user to allocate more power for task offloading. In the sequel, we will show how the user learns to adapt to the environment from the SINR feedbacks.

III-B Computation Model

In this part, how each mobile user $m\in\mathcal{M}$ takes advantage of local execution or computation offloading to satisfy its running applications will be discussed. Without loss of generality, we use $a_{m}(t)$ to quantify the number of task arrivals during slot $t\in\mathcal{T}$, which can be processed starting from slot $t+1$ and is independent and identically distributed (i.i.d) over different time slots with mean rate $\lambda_{m}=\mathbb{E}[a_{m}(t)]$. Besides, we assume that the applications are fine-grained [20]. That is, some bits of the computation tasks denoted by $d_{l,m}(t)$ will be processed on the mobile device, and some other bits denoted by $d_{o,m}(t)$ will be offloaded to and executed by the MEC server. Thus, if $B_{m}(t)$ stands for the queue length of user $m$’s task buffer at the beginning of slot $t$, it will evolve as follows:

[TABLE]

where $B_{m}(0)=0$ and $[x]^{+}=\max(x,0)$.

III-B1 Local computing

In this part, we will show the amount of data bits being processed locally given the allocated local execution power $p_{l,m}(t)\in[0,P_{l,m}]$. To start with, we assume that the number of CPU cycles required to process one task bit at user $m$ is denoted by $L_{m}$, which can be estimated through off-line measurement [46]. By chip voltage adjustment using DVFS techniques [47], the CPU frequency scheduled for slot $t$ can be written by

[TABLE]

where $\kappa$ is the effective switched capacitance depending on the chip architecture. Note that $0\leq f_{m}(t)\leq F_{m}$ with $F_{m}=\sqrt[3]{P_{l,m}(t)/\kappa}$ being the maximum allowable CPU-cycle frequency of user $m$’s device. As a result, the local processed bits at $t$-th slot can be derived by:

[TABLE]

III-B2 Edge computing

To take advantage of edge computing, it is worth noting that the MEC server is usually equipped with sufficient computational resources, e.g., a high-frequency multi-core CPU. Thus, it can be assumed that different applications can be handled in parallel with a negligible processing latency, and the feedback delay is ignored due to the small sized computation output. In this way, all the task data bits offloaded to the MEC server via the BS will be processed. Therefore, according to (16) and given the uplink transmission power $p_{o,m}(t)$, the amount of offloaded data bits of user $m$ during slot $t$ can be derived by

[TABLE]

where $W$ is the system bandwidth and $\gamma_{m}(t)$ is obtained from (16).

IV DRL Based Decentralized Dynamic Computation Offloading

In this section, we will develop a DRL based approach to minimize the computation cost of each mobile user in terms of energy consumption and buffering delay in the proposed multi-user MEC system. Specifically, by employing the DDPG algorithm, a decentralized dynamic computation offloading policy will be learned independently at each user, which selects an action, i.e., allocated powers for both local execution and computation offloading, upon the observation of the environment from its own perspective. It is worth noting that each user has no prior knowledge of the MEC system, which means the number of users $M$, and statistics of task arrivals and wireless channels are unknown to each user agent and thus the online learning process is totally model-free. In the following, by adopting DDPG, the DRL framework for decentralized dynamic computation offloading will be introduced, where the state space, action space and reward function will be defined. Then, how to take advantage of the framework to train and test the decentralized policies is also presented.

IV-A The DRL Framework

State Space: Full observation of the system includes the channel vectors and the queue lengths of the task buffer for all users. However, the system overhead to collect such informations at the BS and then distribute them to each user is very high in practice, which will become even higher as the number of mobile users increases. In order to reduce the overhead and make the MEC system much more scalable, we assume that the state of each user agent is only determined by its local observation of the system, upon which each user will select an action independently from other user agents.

As shown in Fig. 4, at the start of time slot $t$, the queue length of each user $m$’s data buffer $B_{m}(t)$ will be updated according to (17). Meanwhile, one feedback from the BS conveying the last receiving SINR of user $m$ at the BS, i.e., $\gamma_{m}(t-1)$, will be received. At the same time, channel vector $\boldsymbol{h}_{m}(t)$ for the upcoming uplink transmission can be estimated by using channel reciprocity. As a result, from the perspective of each user $m$, the state can be defined as

[TABLE]

where we denote the projected power ratio after ZF detection at the BS for slot $t$ as

[TABLE]

Note that in order to decode user $m$’s symbol without inter-stream interference, ZF detection will project the received signal $\boldsymbol{y}(t)$ to the subspace orthogonal to the one spanned by the other users’ channel vectors [48]. In this way, $\phi_{m}(t)$ can be interpreted as the ratio of unit received power of user $m$’s uplink signal after projection.

Action Space: Based on the current state $s_{m,t}$ of the system observed by each user agent $m$, an action $a_{m,t}$ including the allocated powers for both local execution and computation offloading will be selected for each slot $t\in\mathcal{T}$ as below:

[TABLE]

It is worth noting that, by applying the DDPG algorithm, either power allocation can be elaborately optimized in a continuous action space, i.e., $p_{l,m}(t)\in[0,P_{l,m}]$ and $p_{o,m}(t)\in[0,P_{o,m}]$, to minimize the average computation cost, unlike other conventional DRL algorithms to select from several predefined discrete power levels. Consequently, the high dimension of discrete action spaces can be significantly reduced.

Reward Function: As mentioned in Section II, the behavior of each user agent is reward-driven, which indicates that the reward function plays a key role in the performance of DRL algorithms. In order to learn an energy-aware dynamic computation offloading policy for the proposed MEC model, we consider to minimize the energy consumption while completing tasks within a acceptable buffering delay. Thus, the overall computation cost for each user agent will be counted by both the total energy cost and the penalty on task buffering delay. Notice that it can be known that according to the Little’s Theorem [49], the average queue length of the task buffer is proportional to the buffering delay. In this way, we define the reward function $r_{m,t}$ that each user agent $m$ receives after slot $t$ as

[TABLE]

where $w_{m,1}$ and $w_{m,2}$ are both nonnegative weighted factors, and the reward $r_{m,t}$ is the negative wighted sum of the instantaneous total power consumption and the queue length of task buffer. By setting different values of $w_{m,1}$ and $w_{m,2}$, a tradeoff between energy consumption and buffering delay can be made for the dynamic computation offloading policy. Moreover, it is worth noting that although the DDPG based algorithm maximizes value function of user $m$ starting from the initial state $s_{m,1}$ under policy $\mu_{m}$, i.e.,

[TABLE]

it can be used to approximate of the real expected infinite-horizon undiscounted return [50] at each user agent when $\gamma\rightarrow 1$. That is, the following average computation cost

[TABLE]

will be minimized by applying the learned computation offloading policy $\mu^{*}_{m}$.

IV-B Training and Testing

To learn and evaluate the decentralized computation offloading algorithm, there are two stages including training and testing for the DRL framework. Before training, we need to build the DRL framework with a simulated environment and a group of user agents. The simulated environment is used to generate the training and testing data, which mimics the interaction of the user agents with the MEC system needs to be built, which accepts the decision of each user agent and returns feedbacks of CSI and SINR. For each user agent, the TensorFlow library [51] is used to construct and train the DNNs in the DDPG algorithm.

The detailed training stage is illustrated in Algorithm 1. It is worth noting that the interaction between the user agents and the environment is generally continuing RL tasks [30], which does not break naturally into identifiable episodes. 444Note that for episodic RL tasks, each initial state of the user agent will terminate at a specific state. Thus, in order to have better exploration performance, the interaction of the user agent in the MEC system will be manually start with a random initial state $s_{m,1}$ and terminate at a predefined maximum steps $T_{\max}$ for each episode. At each time step $t$ during an episode, each agent’s experience tuple $(s_{m,t},a_{a,t},r_{m,t},s_{m,t+1})$ will be stored in its own experience buffer $\mathcal{B}_{m}$. Meanwhile, the use agent’s actor and critic network will be updated accordingly using a mini-batch of experience tuples $\{(s_{i},a_{i},r_{i},s^{\prime}_{i})\}_{i=1}^{I}$ randomly sampled from the replay buffer $\mathcal{B}_{m}$. In this way, after the training of $K_{\max}$ episodes, the dynamic computation offloading policy will be gradually and independently learned at each user agent.

As for the testing stage, each user agent will firstly load its actor network parameters learned in the training stage. Then, the user agent will start with an empty data buffer and interact with a randomly initialized environment, after which it selects actions according to the output of the actor network, when its local observation of the environment is obtained as the current state.

V Numerical Results

In this section, numerical simulations will be presented to illustrate the proposed DRL framework for decentralized dynamic computation offloading in the MEC system. System setup for the simulations is firstly introduced. Then, performance of the DRL framework is demonstrated and compared with some other baseline schemes in the scenarios of single user and multiple users, respectively.

V-A Simulation Setup

In the MEC system, time is slotted by $\tau_{0}=1\mathrm{ms}$. For the beginning of every episode, each user $m$’s channel vector is initialized as $\boldsymbol{h}_{m}(0)\sim\mathcal{CN}(0,h_{0}(d_{0}/d_{m})^{\alpha}\boldsymbol{I}_{N})$, where the path-loss constant $h_{0}=-30\mathrm{dB}$, the reference distance $d_{0}=1\mathrm{m}$, the path-loss exponent $\alpha=3$, and $d_{m}$ is the distance of user $m$ to the BS in meters. In the following slots, $\boldsymbol{h}_{m}(t)$ will be updated according to (15), where the channel correlation coefficient $\rho_{m}=0.95$ and the error vector $\boldsymbol{e}(t)\sim\mathcal{CN}(0,h_{0}(d_{0}/d)^{\alpha}\boldsymbol{I}_{N})$ with $f_{d,m}=70\mathrm{Hz}$. Additionally, we set the system bandwidth to be $1\mathrm{MHz}$, the maximum transmission power $P_{o,m}=2\mathrm{W}$, and the noise power $\sigma_{R}^{2}=10^{-9}\mathrm{W}$. On the other hand, for local execution, we assume that $\kappa=10^{-27}$, the required CPU cycles per bit $L_{m}=500$ cycles/bit, and the maximum allowable CPU-cycle frequency $F_{m}=1.26\mathrm{GHz}$, from which we know that the maximum power required for local execution $P_{l,m}=2\mathrm{W}$.

To implement the DDPG algorithm, for each user agent $m$, the actor network and critic network is a four-layer fully connected neural network with two hidden layers. The number of neurons in the two hidden layers are $400$ and $300$, respectively. The neural networks use the Relu, i.e., $f(x)=\max(0,x)$, as the activation function for all hidden layers, while the final output layer of the actor uses a sigmoid layer to bound the actions. Note that for the critic, actions are not included until the second hidden layer of the Q-network. Adaptive moment estimation (Adam) method [52] is used for learning the neural network parameters with a learning rate of $0.0001$ and $0.001$ for the actor and critic respectively. The soft update rate for the target networks is $\tau=0.001$. To initialize the network layer weights, settings in experiment of [39] is adopted. Moreover, in order to explore well, the Ornstein-Uhlenbeck process [53] with $\theta=0.15$ and $\sigma=0.12$ is used to provide temporal correlated noise. The experience replay buffer size is set as $|\mathcal{B}_{m}|=2.5\times 10^{5}$.

With slight abuse of notation, we introduce a tradeoff factor $w_{m}\in[0,1]$ for each use agent $m\in\mathcal{M}$, which is used to represent the two nonnegative weighted factors by $w_{m,1}=10w_{m}$ and $w_{m,2}=1-w_{m}$. Thus, the reward function $r_{m,t}$ in (24) can be written by

[TABLE]

from which we can make a tradeoff between energy consumption and buffering delay by simply setting a single factor $w_{m}$. Moreover, the number of episodes is $K_{\max}=2000$ in the training stage, and the maximum steps of each episode is $T_{\max}=200$.

For comparison, the baseline strategies are introduced as follows:

V-A1 Greedy Local Execution First (GD-Local)

For each slot, the user agent firstly attempts to execute task data bits locally as many as possible. Then, the remaining buffered data bits will be offloaded to the MEC.

V-A2 Greedy Computation Offloading First (GD-Offload)

Similar to the GD-local strategy, each user agent firstly makes its best effort to offload data bits to the MEC, and then local execution will be adopted to process the remaining buffered data bits for each time slot.

V-A3 DQN based Dynamic Offloading (DQN)

To evaluate performance of the proposed DDPG base algorithm, the conventional discrete action space based DRL algorithm, i.e., DQN [31], is also implemented for the dynamic computation offloading problem. Specifically, for each user $m$, the power levels for local execution and computation offloading are defined as $\mathcal{P}_{l,m}=\{0,\frac{P_{l,m}}{L-1},\dots,P_{l,m}\}$ and $\mathcal{P}_{o,m}=\{0,\frac{P_{o,m}}{L-1},\dots,P_{o,m}\}$, where the number of power levels is set as $L=8$. Thus, the action space for each user agent to select from is $\mathcal{P}_{l,m}\times\mathcal{P}_{o,m}$. Besides, $\epsilon$-greedy exploration and Adam method are adopted for training.

V-B Single User Scenario

In this part, numerical results of training and testing for the single user scenario are illustrated. The user is assumed to be randomly located in a distance of $d_{1}=100$ meters to the BS.

V-B1 Training

As shown in Fig. 5(a) and Fig. 5(b), the training process of the single-user dynamic computation offloading is presented by setting $w_{1}=0.5$ and $w_{1}=0.8$, respectively. Note that the results are averaged from $10$ runs of numerical simulations. In each figure, we will compare two different cases, where the task arrival rate is set as $\lambda_{1}=2.0\mathrm{Mbps}$ and $\lambda_{1}=3.0\mathrm{Mbps}$, respectively. It can be observed that for both policies learned from DDPG and DQN, the average reward of each episode increases as the interaction between the user agent and the MEC system environment continues, which indicates that efficient computation offloading policies can be successfully learned without any prior knowledge. Besides, the performance of each learned policy becomes stable after about $1500$ episodes. On the other hand, performance of the policy learned from DDPG is always better than DQN for different scenarios, which demonstrates that for continuous control problems, DDPG based strategies can explore the action space more efficiently than DQN based strategies.

V-B2 Testing

In the training stage, we have obtained dynamic computation offloading policies by applying the DDPG based and DQN based learning algorithms after $K_{\max}=2000$ episodes, respectively. For different task arrival rates ranging from $\lambda_{1}=1.5\sim 4.0\mathrm{Mbps}$, the actor and critic network will be trained with the same network architecture and hyper-parameters. To compare the performance of different policies, testing results are averaged from $100$ runs of numerical simulations, where each run consists of $10000$ steps. As shown in Fig. 6 and Fig. 7, testing results are presented for $w_{1}=0.5$ and $w_{1}=0.8$, respectively, each of which includes the performance of average reward, power consumption and buffering delay. It can be observed from Fig. 6 that the average reward will increase as the task arrival rate grows, which indicates that the computation cost is higher for a larger computation demand. Specifically, the increased computation cost results from a higher power consumption and a longer buffering delay. Moreover, although the DDPG based strategy outperforms both greedy strategies with minimum average reward, which, however, slightly compromises the buffering delay to achieve the lowest energy consumption. It is worth noting that the average reward of the DQN based strategy is higher than the greedy strategies, which is due to the limited number of discrete power levels of DQN based strategy555Note that although finer grained discretization of the action space will potentially lead to better performance, the number of actions increases exponentially with the number of degrees of freedom, which makes it much more challenging to explore efficiently and in turn significantly decreases the performance..

In Fig. 7, testing results for a larger tradeoff factor $w_{1}=0.8$ are also provided. From (29), we know that a larger $w_{1}$ will give more penalty on power consumption in the reward function, i.e., the computation cost. In this scenario, the average reward of the DDPG based strategy outperforms all the other strategies, and the gap is much larger than that in the case of $w_{1}=0.5$. Specifically, this is achieved by much lower power consumption and increased buffering delay.

V-B3 Power-Delay Tradeoff

We also investigate testing results for the power-delay tradeoff by setting different values of $w_{1}$ in Fig. 8. It can be inferred from the curves that, there is a tradeoff between the average power consumption and the average buffering delay. Specifically, with a larger $w_{1}$, the power consumption will be decreased by sacrificing the delay performance, which indicates that in practice $w_{1}$ can be tuned to have a minimum power consumption with a given delay constraint. It is also worth noting noting that for each value of $w_{1}$, the policy learned form DDPG always has better performance in terms of both power consumption and buffering delay, which demonstrates the superiority of the DDPG based strategy for continuous power control.

V-C Multi-User Scenario

In this part, numerical results for the multi-user scenario is presented. There are $M=3$ mobile users in the MEC system, each of which is randomly located in a distance of $d_{m}=100$ meters to the BS, and the task arrival rate is $\lambda_{m}=m\times 1.0\mathrm{Mbps}$, for $m\in\{1,2,3\}$.

V-C1 Training

By setting $w_{m}=0.5$ for all the users, the training process has been shown in Fig. 9. It can be observed that for each mobile user, the average reward increases gradually when the mobile user interacts with the MEC system after more episodes. Thus, we know that for both the DDPG based and DQN based strategies, efficient decentralized dynamic computation offloading policies can be learned at each mobile user, especially for heterogeneous users with different computation demands. Moreover, it can be inferred that the higher computation cost needs to be paid by the user with a higher computation demand. Meanwhile, compared with the single user scenario, the average reward obtained in the multi-user scenario is much lower for the same task arrival rate. It is due to the fact that the spectral efficiency of data transmission will be degraded when more mobile users are served by the BS. Hence, more power will be consumed in computation offloading in the multi-user scenario.

V-C2 Testing

By loading the neural network parameters learned by the DDPG based and DQN based algorithms after $K_{\max}=2000$ episodes, testing results of different dynamic computation offloading policies are compared in Table II and Table III. From Table II, we know that the average rewards of user $2$ and user $3$ adopting DDPG based strategies are better than all other strategies under the scenario of $w_{m}=0.5$. However, as for user $1$, the DDPG based strategy is slightly worse than the GD-Local strategy, which indicates that the exploration of DDPG for a small allocated power needs to be further improved. Also, it can be observed that both the DDPG based and DQN based strategies can achieve much lower power consumption with a little compromised buffering delay.

By setting the tradeoff factor $w_{m}=0.8$ as shown in Fig. III, the DDPG based strategies obtain the best average reward at each user agent. Moreover, the performance gaps between the DDPG based strategies and other greedy strategies become larger. Besides, with more penalty given to the power consumption, the consumed power of each user is much lower than that of $w_{m}=0.5$, which, however, results in a moderately increased buffering delay. Thus, we know that for the multi-user scenario, power consumption can be minimized with a satisfied average buffering delay, by selecting a proper value of $w_{m}$. Notice that again the DDPG based strategies outperform the DQN based strategies in terms of average reward for all users.

VI Conclusion

In this paper, we considered a multi-user MEC system, where tasks arrive stochastically and wireless channels are time-varying at each user. In order to minimize the long-term average computation cost in terms of power consumption and buffering delay, the design of DRL based decentralized dynamic computation offloading algorithms has been investigated. Specifically, by adopting the continuous action space based DRL approach named DDPG, an efficient computation offloading policy has been successfully learned at each mobile user, which is able to allocate powers of local execution and task offloading adaptively from its local observation of the MEC system. Numerical simulations have been performed to verify the superiority of the proposed DDPG based decentralized strategy over the conventional DQN based discrete power control strategy and some other greedy strategies with reduced computation cost. Besides, the power-delay tradeoff for both the DDPG based and DQN based strategies has been also studied.