Learning Deterministic Policy with Target for Power Control in Wireless Networks

TL;DR

This paper introduces a Deep Reinforcement Learning framework with deterministic policy and target for power control in wireless networks, significantly improving energy efficiency and throughput amid complex interference patterns.

Contribution

It presents a novel DRL-DPT framework that effectively handles continuous state and action spaces, ensuring convergence and outperforming existing ICIC schemes.

Findings

Up to 15% improvement in energy efficiency.

Faster convergence rate compared to traditional methods.

Consistent performance across various interference scenarios.

Abstract

Inter-Cell Interference Coordination (ICIC) is a promising way to improve energy efficiency in wireless networks, especially where small base stations are densely deployed. However, traditional optimization based ICIC schemes suffer from severe performance degradation with complex interference pattern. To address this issue, we propose a Deep Reinforcement Learning with Deterministic Policy and Target (DRL-DPT) framework for ICIC in wireless networks. DRL-DPT overcomes the main obstacles in applying reinforcement learning and deep learning in wireless networks, i.e. continuous state space, continuous action space and convergence. Firstly, a Deep Neural Network (DNN) is involved as the actor to obtain deterministic power control actions in continuous space. Then, to guarantee the convergence, an online training process is presented, which makes use of a dedicated reward function as the target rule and a policy gradient descent algorithm to adjust DNN weights. Experimental results show that the proposed DRL-DPT framework consistently outperforms existing schemes in terms of energy efficiency and throughput under different wireless interference scenarios. More specifically, it improves up to 15% of energy efficiency with faster convergence rate.

Figures

Tables

Table 1. TABLE I: Experimental parameters.

Traffic Model	Full Buffer Model
Transmission Power	30dbm(SBSs)
System Bandwidth	10MHz
SBS-user Path Loss Model	$140.7+26.7lo{g}_{10}\left(d\right)$
Cell Layer	$2\times 2$ or $5\times 5$ Grid Model
Distance Between SBSs	50m
Learning Rate	0.1
$\gamma$	0.5

Equations

$$Q^{\pi }(s_t,a_t)=\mathbb{E}[R_t|s_t,a_t]$$

$$Q^{\pi }(s_t,a_t)=\mathbb{E}[r(s_t,a_t)+\gamma \mathbb{E}{Q}^{\pi }(s_{t+1},a_{t+1})]$$

$$L(\theta )=\mathbb{E}[(Q(s_t,a_t|\theta )-y_t{)}^2]$$

$$C(\theta )\propto (1/R_t)$$

$$S_{m,n,l}=P_{m,l}\cdot g_{m,n,l}$$

$$SIN{R}_{m,n,l}=\frac{P_{m,l}{g}_{m,n,l}}{{\sigma }_{m,n,l}^2+{\sum }_{1\le k\le M,k\ne m}{P}_{k,l}{g}_{k,n,l}}$$

$$R_{m,n}=\frac{1}{L{T}_{sf}}\sum _{l=1}^L{T}_{sf}Wlog(1+SIN{R}_{m,n,l})$$

$$E_{m,l}=△p{P}_{m,l}+p_0,0\le P_{m,l}\le P$$

$$\eta =\frac{{\sum }_{m=1}^M{\sum }_{n=1}^{N_m}{R}_{m,n}}{{\sum }_{m=1}^M{\sum }_{l=1}^L{E}_{m,l}}$$

$$\max$$

$$\text{s}.t.$$

$$0\le P_{m,l}\le P,\forall m\in \mathbb{M},\forall l\in \mathbb{L}$$

$$S_m=$$

$$...,\sum _{n=1}^{N_m}{S}_{m-1,n},\sum _{n=1}^{N_m}{S}_{m+1,n},...,\sum _{n=1}^{N_m}{S}_{M,n}\}$$

$$S_m=\{{\rho }_{m,1},{\rho }_{m,2},...,{\rho }_{m,k},...,{\rho }_{m,M}\}$$

$$\text{S}=\left[\begin{array}{ccccc}{\rho }_{1,1}& {\rho }_{1,2}& {\rho }_{1,3}& ...& {\rho }_{1,M}\\ {\rho }_{2,1}& {\rho }_{2,2}& {\rho }_{2,3}& ...& {\rho }_{2,M}\\ ...& ...& ...& ...& ...\\ {\rho }_{M,1}& {\rho }_{M,2}& {\rho }_{M,3}& ...& {\rho }_{M,M}\end{array}\right]$$

$$H_{1i}(m,f,l)=\sum _{k=1}^M{h}_1(k,f,l)\text{S}(m,k)$$

$$H_{1o}(m,f,l)=act[H_{1i}(m,f,l)]$$

$$1\le m\le M,1\le f\le F,1\le l\le L$$

$$H_{2i}(m,l)=\sum _{f=1}^F{h}_2(m,f,l)H_{1o}(m,f,l)$$

$$H_{2o}(m,l)=act[H_{2i}(m,l)]$$

$$H_{3i}(m,l)=\sum _{k=1}^M{h}_3(k,m,l)H_{2o}(m,l)$$

$$H_{3o}(m,l)=act[H_{3i}(m,l)]$$

$$\text{P}=\left[\begin{array}{ccccc}{P}_{1,1}& P_{1,2}& P_{1,3}& ...& P_{1,L}\\ P_{2,1}& P_{2,2}& P_{2,3}& ...& P_{2,L}\\ ...& ...& ...& ...& ...\\ P_{M,1}& P_{M,2}& P_{M,3}& ...& P_{M,L}\end{array}\right]$$

$$r=\gamma exp(\eta )$$

$$C=-r$$

$${\delta }_{h3}(m,l)=\frac{\partial C}{\partial H_{3o}(m,l)}\frac{\mathrm{d}act}{\mathrm{d}{h}_{3i}(m,l)}$$

$${\delta }_{h2}(m,l)=\sum _{k=1}^M[{\delta }_{h3}(k,l)h_3(m,k,l)]\frac{\mathrm{d}act}{\mathrm{d}{H}_{2i}(m,l)}$$

$${\delta }_{h1}(m,f,l)={\delta }_{h2}(m,l)h_2(m,1,f,l)\frac{\mathrm{d}act}{\mathrm{d}{H}_{1i}(m,f,l)}$$

$${\nabla }_{h_3(m_1,m_2,l)}C={\delta }_{h3}(m_2,l)H_{2o}(m_1,l)$$

$${\nabla }_{h_2(m,1,f,l)}C={\delta }_{h2}(m,l)H_{1o}(m,f,l)$$

$${\nabla }_{h_1(1,m,f,l)}C=\sum _{k=1}^M[{\delta }_{h1}(k,f,l)I(k,m)]$$

$$h_3^{\prime }=h_3-lr{\nabla }_{h_3}C$$

$$h_2^{\prime }=h_2-lr{\nabla }_{h_2}C$$

$$h_1^{\prime }=h_1-lr{\nabla }_{h_1}C$$

Full text

I Introduction

Reinforcement learning (RL) is provided with the ability of learning a rewarding behavior in a pre-unknown environment via trial-and-error [3]. It has been successfully applied in many physical tasks. Recently, significant advances in RL have been obtained by combining deep learning (DL) into RL, resulting in deep reinforcement learning (DRL), mainly represented by the algorithms of “Deep Q Network” (DQN) [4] and “Deep Deterministic Policy Gradient” (DDPG) [5]. DRL is capable of human-level performance by using deep neural network function approximators.

In modern wireless networks, especially where small cell networks are densely deployed, the interferences between co-frequency cells are getting heavier and heavier, making inter-cell interference coordination (ICIC) essential to improve the system performance. While traditional optimization-based ICIC schemes are hand-crafted and suffer degradation with complex interference patterns [20], RL is promising to achieve the breakthrough [12]. However, there are several challenges to apply RL in wireless networks. First, the state and action spaces in wireless networks are uncountable, making the tabular-represented Q learning [13] and the action-discrete DQN infeasible. Both of them base on the iteration of Q-value, which limits the problem space. The observations of wireless networks range in a large scale, varying with the distribution of users and the dynamic power allocation. And the action space, which refers to power allocations in ICIC, is continuous because of the finer-grained requirements [14]. Although DDPG has been proposed to solve the continuous control problems, it suffers with the slow convergence due to the neural networks approximated actor-critic model. All of them are value-based, which means they are impossible to learn fast since they have to update the values by large amounts of iterations. Also they may suffer from oscillations when converge. Another interesting thing is that a lot of excellent analyses about wireless networks have been done and they may provide guidance to the learning methods as something like “teacher’s advice”. While the existed RL algorithms cannot leverage the well-known prior knowledge obviously.

In this paper, we propose a novel deep reinforcement learning with deterministic policy and target (DRL-DPT) framework for ICIC in wireless networks. It is shown that the energy-efficiency of different wireless networks improves greatly by using DRL-DPT. In fact, DRL-DPT could obtain up to 15% increase in terms of the energy-efficiency and the system throughput compared to baselines. DRL-DPT uses a deep neural network (DNN) to play the role of actor underlying the success of DDPG to deal with the continuous state and action spaces. DRL-DPT is policy-based and do not rely on the update of Q-values. It gives deterministic power allocation schemes at each iteration without any extra computation. It also adapts the mature theories in wireless networks to help build the optimization rules, aiming at higher energy-efficiency [6]. A dedicated reward function is designed to guide the learning process based on the deterministic policy gradient (DPG) algorithm [7]. By removing the critic network and learning directly from the expected objective, DRL-DPT guarantees the convergence with an acceptable speed, as well as good stability. A simple compare between Q learning, DQN, DDPG and DRL-DPT can be found in fig. 1.

The rest of this paper is organized as follows. Section 2 briefly introduces some backgrounds on RL and DRL. Section 3 introduces the considering application scenario and some useful acknowledgements. In section 4, illustrations about the proposed DRL-DPT are given in detail. Section 5 describes the experimental results. Finally in section 6, we conclude our work with discussion about future work.

II Background

RL is a process where an agent interacts with its environment, receiving observations and selecting actions to maximize a reward signal provided by the environment. It has made many achievements in robotic tasks. Recently, some literatures also adapt RL to communication systems.

RL algorithms typically leverage the Markov decision process (MDP) formulation. In an MDP, $\mathbb{A}$ is a set of actions and $\mathbb{S}$ is a set of states. There are two functions within this process: a transition function $(T\!:\mathbb{S}\!\times\!\mathbb{A}\!\mapsto\!\mathbb{S})$ and a reward function $(r\!:\mathbb{S}\!\times\!\mathbb{A}\!\mapsto\!r)$. The return from a state is often described as the sum of discounted future rewards: $R_{t}\!=\!\sum_{i=t}^{T}\gamma^{i-t}r_{t}(s_{i},a_{i})$, where $\gamma\!\in\![0,1]$ is a discount factor.

The action-value function is used in many RL algorithms to describe the expected return after taking an action $a_{t}$ in state $s_{t}$ following policy $\pi$:

[TABLE]

Many approaches in RL make use of the recursive relationship known as the Bellman equation:

[TABLE]

It is possible to learn the optimal policy $\pi^{*}$ using transitions from a different stochastic behavior policy. Different RL algorithms have different ways to learn the optimal policy which maximizing the reward. Q Learning, a widely used off-policy algorithm, uses the greedy policy: $\pi^{*}(s)=argmax_{a}Q(s,a)$. In DQN and DDPG, function approximators parameterized by $\theta$ are used, and they optimize by minimizing the loss:

[TABLE]

where $y_{t}=r(s_{t},a_{t})+\gamma Q(s_{t+1},\pi^{*}(s_{t+1})|\theta)$ is calculated by another target network.

In this paper, we denote the discount factor to be 0 since the considered ICIC problem is some how instant. We approximate the policy by a neural network, too. But when optimizing, we directly aim at maximizing the feedback reward $R_{t}$ by formulating the reward function theoretically. In practice, the optimization is done by minimizing the cost $C(\theta)$:

[TABLE]

III System Model

In order to describe the proposed framework clearly, we demonstrate the scenario and assumptions briefly in this section. As shown in Fig. 2, modern communication systems often overlay the small base stations (SBSs) based on the orthogonal macrocell frequency to enlarge the system capacity, however, such practices will bring heavy inter-cell interferences [16]. To handle such a challenge, this paper proposes DRL-DPT algorithm to optimize inter-cell interference coordination by doing. Our algorithm runs on mobile edge computing (MEC) server [17].

To depict the inter-cell interference mathematically, we assume $M$ SBSs and $N_{m}$ users connected to SBS $m$. At time slot $l$, user $n$ gets interested signal from its associated SBS $m$, the signal strength can be described as

[TABLE]

where $P_{m,l}$ represents the transmission power of SBS $m$ at time slot $l$, $g_{m,n,l}$ represents the channel gain from SBS $m$ to user $n$.

Similarly, user $n$ receives interference signals $S_{k,n,l}(k\!\not=\!m)$ from other SBSs. According to Shannon theorem, the signal to interference and noise ratio (SINR) and downlink data rate of user $n$ can be expressed as

[TABLE]

[TABLE]

where $\sigma_{m,n,t}^{2}$ is the noise. $L$ represents the length of a frame and $T_{sf}$ is the duration of a subframe. $W$ represents the bandwidth assigned to user $n$.

The system power consumption is evaluated by the power model introduced in [6], which is expressed as

[TABLE]

where $\bigtriangleup p$ is a parameter related to power amplifier. $p_{0}$ represents the circuit power and $P$ stands for the maximum power the SBSs could choose. The system performance is graded by energy efficiency, which is defined as the ratio of system throughput and system power consumption [18]:

[TABLE]

To implement ICIC and enhance the energy efficiency, the DRL-DPT algorithm does power control to schedule the interference signals. It observes the system and gives specific power allocation for every SBS at each frame. Detailed process can be found in the next section.

IV Proposed Algorithm

In this section, DRL-DPT is illustrated in detail. Firstly, we formulate the optimization objective based on the widely used theories in wireless networks. It provides DRL-DPT with a learning target. Then in the next subsection, we present and explain the structure of DRL-DPT. Finally, we demonstrate the learning mechanism guided by the target rule and account for the no-label gradient descent executed on the actor.

IV-A Target

In this paper, we focus on maximizing the system energy efficiency to trade off throughput and power consumption. The optimization problem can be formulated as

[TABLE]

where $R_{n}^{th}$ refers to the throughput requirement of user $n$. Respectively, $\mathbb{M}\!=\!\{1,2,...,M\},\mathbb{N}_{m}\!=\!\{1,2,...,N_{m}\},\mathbb{L}\!=\!\{1,2,...,L\}$. (10b) indicates the demand that users’ requirements have to be satisfied. (10c) means the transmission powers are positive numbers less than the predefined upper limit. When designing the modules of DRL-DPT, we consider the maximization objective as well as two constraints.

IV-B Neural Network Approximated Actor

The structure of the proposed DRL-DPT is shown in Fig. 3.

It can be divided into $L$ drawers with same architecture transversely. Every drawer corresponds to a subframe while all of them share the same input. This parted design is inspired by the fairness between SBSs and could reduce the complexity of DRL-DPT. There are four layers in DRL-DPT, including input layer, single feature extracting layer, global feature extracting layer and output layer. Their forward propagation functions are described in the following paragraphs.

1. (1)

Input layer: At the beginning of each frame, the SBSs collect reference signals from their associated users to obtain necessary information about the wireless system. To reduce the data transmitted to the MEC server and to reduce the size of DRL-DPT, some pre-operations are manipulated at SBSs. The results can be regarded as partial observations and have the following expressions.

[TABLE]

To simplify the presentation, we rewrite the expression as

[TABLE]

where \rho_{m,k}=\left\{\begin{array}[]{lr}\sum_{n=1}^{N_{m}}R_{n}^{th},k\!=\!m\\ \sum_{n=1}^{N_{m}}S_{k,n},k\!\not=\!m\end{array}\right.. $S_{m}$ contains the information about the required load and the suffered interferences of SBS $m$.

The server receives these partial observations and integrates them as the final state input to DRL-DPT. The integrated state is expressed as

[TABLE]

The input state is organized in the form of a $M\!\times\!M$ matrix and its diagonal elements represent the load of each SBS while its non-diagonal elements represent interferences from other SBSs. 2. (2)

Single feature extracting layer: $F\!\times\!L$ filters with size of $1\!\times\!M$ are set up to extract the feature about the relative strength between different interferences as well as their influence to the specific SBS’s load. $F$ is the number of filters set in one drawer. The filters in this layer has the same size as the mentioned partial states, so they can abstract input on SBS level and obtain a general impression. The inputs of the $M\!\times\!F\!\times\!L$ nodes in this layer are:

[TABLE]

$h_{1}(k,f,l)$ is the $k$th weight of the $f$th filter in the $l$th drawer in this layer. And the outputs are

[TABLE]

where $act(\cdot)=sigmoid(\cdot)$ is the activation function we choose. 3. (3)

Global feature extracting layer: There are $F\!\times\!L$ filters with size of $M\!\times\!1$ in this layer. They do comparisons between the extracted features of different SBSs to get details so that DRL-DPT can schedule the system wisely. The $M\!\times\!L$ nodes’ inputs in this layer are expressed as

[TABLE]

where $h_{2}(m,f,l)$ is the $m$th weight of the $f$th filter in the $l$th drawer in the global feature extracting layer. Their outputs are

[TABLE] 4. (4)

Output layer: Partial full-connections are operated to get the final output actions in this layer. The output layer has $M\!\times\!M\!\times\!L$ weights and $M\!\times\!L$ nodes in total. There are similar inputs and outputs in this layer as the former two:

[TABLE]

[TABLE]

$h_{3}(k,m,l)$ is the $(k\!\times\!m)$th weight in the $l$th drawer. Actually, the output of DRL-DPT is the precise power allocation scheme, which can be expressed as

[TABLE]

where $P_{m,l}\!=\!P\!\cdot\!H_{3o}(m,l)$.

Finally, DRL-DPT gets a $M\!\times\!L$ matrix which represent the learned optimal power allocations. The MEC server informs SBSs with the corresponding elements in the output power matrix to realize ICIC by power control.

IV-C Learning Mechanism

We have described the forward propagation process of the actor above. In this section, we explain the learning mechanism of DRL-DPT and give theoretical supports. The key point is the design of the reward function. In typical RL algorithms, the reward function is usually regarded as unknown, making the learning process slow and uncertain. However, in many scenarios, such as the considered wireless system, a lot of mature analyses and effective works can be found. It is meaningful to seek guidance from them. In this paper, we develop a deterministic target based on the common knowledge in wireless networks and give the theoretical presentation of reward function. Then a deterministic policy gradient descent algorithm is employed to adjust the parameterized policy, aiming at maximizing the reward. The dedicated reward function is consistent with the objective in equation (10). It saves the agent from conjecturing the reward, allowing more direct and quick learning. The detailed process is illustrated as follow:

1. (1)

Reward function: With the target of maximizing the system energy efficiency, we denote the reward function theoretically on the bases mentioned in section 3. The expression is

[TABLE]

where $\gamma$ is a positive constant coefficient. For specific state S and action P, $r$ could be calculated.

This design provides a certain representation for the rewards and it is related to the output actions. It determines the learning target clearly for the actor as well as indicating the learning approach. By using a policy gradient descent algorithm, the actor tunes its parameters iteratively to get higher rewards. The maximum could be obtained after enough iterations, meaning that the system is optimized in term of energy efficiency. This well-defined reward function essentially simplify the learning process by saving the agent from uncertain predictions. It provides a deterministic target which is in line with the optimization objective. It also shows the rules of action selecting so that the actor may know what to do more explicitly. 2. (2)

Policy gradient descent: To train the actor under the guidance of the interactive reward function means to maximize the reward $r$. In practice, we minimize the cost:

[TABLE]

A deterministic policy gradient descent algorithm with error back propagation [19] is adopted. When a reward is fed back to DRL-DPT and converted to cost, the error of each layer is computed by

[TABLE]

Then the policy gradients of every weights in the actor network can be computed by

[TABLE]

Assuming the learning rate as $lr$, the weights are adjusted by the following rules:

[TABLE]

In this way, the actor performs a self-learning towards the target rule determinately.

The entire procedure of using DRL-DPT to implement ICIC in wireless networks is concluded as algorithm 1.

$\epsilon$ is a predefined threshold indicated the termination of the algorithm.

V Experiments

Experiments are performed to checkout the proposed DRL-DPT algorithm in terms of effectiveness and maneuverability. Some experimental parameters are given in Tab. I.

In Fig. 4, the system performance under different ICIC schemes are displayed. 25 SBSs are deployed according to the grid model and the number of users varies. We choose a traditional time-domain ICIC scheme based on ABS [20] and a power control scheme without ICIC as two baselines. As expected, the power control scheme without ICIC shows the worst system performance in terms of energy efficiency and throughput. It indicates that ICIC is necessary for wireless networks with densely deployed SBSs. Fig. 4 also shows that the proposed learning scheme outperforms the traditional ABS based schemes up to 15% in terms of energy efficiency. As the number of users increases from 50 to 200, more performance gains are obtained with DRL-DPT. However, in simulation scenarios with 250 and 300 users, the system performance almost remains constant for all schemes. The reason is that the system runs into resource saturation, which means that there is no power available for power control operation.

In order to show the convergence of DRL-DPT, a simple DDPG framework with a three-layer critic is also carried out to compare with. The critic predicts the system reward and guides the actor when selecting actions. Meanwhile, the critic also trains itself according to the actual feedback rewards to realize accurate predictions. Fig. 5 shows the convergence of DRL-DPT and DDPG under different system scales. Fig. 5(a) shows the result when there are 20 users connected to the network. DRL-DPT improves the energy-efficiency quickly and gets stable in about 20 iterations, while DDPG learns slowly. When there are more users in the system, i.e. 100 users considered in fig. 5(b), DRL-DPT can still converge within tens of iterations and stay stable. And under this condition, DDPG becomes more unstable and suffers from violent oscillation. These results reveal the significance of deterministic targets. DRL-DPT learns with the guidance of the deterministic target rules so that it behaves with a better sense of direction. To the contrary, DDPG has to predict the rules at first, so it takes much more time to find the optimal actions and suffers instability. With the pre-defined target rules, DRL-DPT can find the rewarding behavior quickly and accurately.

VI Conclusion

In this paper, we propose a novel deep reinforcement learning algorithm to implement the inter-cell interference coordination in DSCNs. The proposed algorithm obtains good grades working in continuous domain with deterministic policy and target rule. Some mature acknowledgements in wireless networks are adapted to DRL to implement a theoretical guidance. Though the work is still primary and superficial, it is believed that RL technologies will bring huge energy to the future wireless networks. And it is promising that our research could inspire more and more excellent works.

As future work, we will improve the framework to better adapt to the dynamics in wireless networks. It is also valuable to find ways to reduce the system burden further.

ACKNOWLEDGMENT

This work was supported in part by the National Science Foundation of China (No.91538203, 61390513, 61771445) and the Fundamental Research Funds for the Central Universities.