Learning to Optimize: Training Deep Neural Networks for Wireless Resource Management

TL;DR

This paper introduces a deep learning approach to approximate complex wireless resource management algorithms, enabling near real-time performance with high accuracy and significant speed improvements over traditional optimization methods.

Contribution

It provides a theoretical framework for learning optimization algorithms with DNNs and demonstrates practical efficiency in approximating power allocation algorithms like WMMSE.

Findings

DNNs can accurately approximate WMMSE with mild error dependence on network size.

DNN-based methods achieve orders of magnitude faster computation.

Theoretical analysis confirms learnability of certain optimization algorithms.

Abstract

For the past couple of decades, numerical optimization has played a central role in addressing wireless resource management problems such as power control and beamformer design. However, optimization algorithms often entail considerable complexity, which creates a serious gap between theoretical design/analysis and real-time processing. To address this challenge, we propose a new learning-based approach. The key idea is to treat the input and output of a resource allocation algorithm as an unknown non-linear mapping and use a deep neural network (DNN) to approximate it. If the non-linear mapping can be learned accurately by a DNN of moderate size, then resource allocation can be done in almost real time -- since passing the input through a DNN only requires a small number of simple operations. In this work, we address both the thereotical and practical aspects of DNN-based algorithm approximation with applications to wireless resource management. We first pin down a class of optimization algorithms that are `learnable' in theory by a fully connected DNN. Then, we focus on DNN-based approximation to a popular power allocation algorithm named WMMSE (Shi {\it et al} 2011). We show that using a DNN to approximate WMMSE can be fairly accurate -- the approximation error $\epsilon$ depends mildly [in the order of $\log(1/\epsilon)$] on the numbers of neurons and layers of the DNN. On the implementation side, we use extensive numerical simulations to demonstrate that DNNs can achieve orders of magnitude speedup in computational time compared to state-of-the-art power allocation algorithms based on optimization.

Figures

Tables

Table 1. TABLE IV: Sum-Rate and Computational Performance for IMAC (With Different Cell Radius R 𝑅 R )

# of base stations	sum-rate performance				computational time performance
and users (N,K)	R=100m	R=200m	R=300m	R=500m	R=100m	R=200m	R=300m	R=500m
$(3,12)$	98.30%	99.51%	99.62%	98.91%	7.78%	19.16%	19.64%	19.77%
$(3,18)$	97.45%	98.27%	97.80%	95.58%	4.58%	13.06%	14.67%	13.34%
$(3,24)$	96.82%	96.77%	93.83%	87.32%	2.81%	10.68%	12.44%	10.83%
$(7,28)$	94.53%	97.19%	95.41%	90.22%	1.58%	5.84%	8.68%	8.69%
$(20,80)$	90.36%	86.27%	65.55%	43.48%	0.61%	1.75%	3.21%	3.31%

Equations

$$\text{mbox}{sinr}_k\triangleq \frac{|h_{kk}{|}^2{p}_k}{{\sum }_{j\ne k}|h_{kj}{|}^2{p}_j+{\sigma }_k^2},$$

$$\begin{array}{rl}\underset{p_1,\dots ,p_K}{\max }& \sum _{k=1}^K{\alpha }_k\log \left(1+\frac{|h_{kk}{|}^2{p}_k}{{\sum }_{j\ne k}|h_{kj}{|}^2{p}_j+{\sigma }_k^2}\right)\\ \text{s.t.}& 0\le p_k\le P_{\max },\forall k=1,2,\dots ,K,\end{array}$$

$$\begin{array}{rl}\underset{\{w_k,u_k,v_k{\}}_{k=1}^K}{\min }& \sum _{k=1}^K{\alpha }_k\left(w_k{e}_k-\log (w_k)\right)\\ \text{s.t.}& 0\le v_k\le \sqrt{P_k},k=1,2,\dots ,K.\end{array}$$

$$e_k=(u_k|h_{kk}|v_k-1{)}^2+\sum _{j\ne k}(u_k|h_{kj}|v_j{)}^2+{\sigma }_k^2{u}_k^2$$

$$\underset{x\in X}{\min }f(x)=(x^2-z{)}^2$$

$$x^{t+1}=x^t-\alpha \nabla f(x^t)=x^t-2\alpha x^t((x^t{)}^2-z),t=0,\cdots ,T$$

$$x^{t+1}=f^t(x^t,z),t=0,1,2,\dots ,T$$

$$x^T=f^T(f^{T-1}(\dots f^1(f^0(x^0,z),z)\dots ,z),z)\triangleq F^T(x^0,z)$$

$$\underset{(x^0,z)\in X\times Z}{\sup }\Vert NE{T}_N(x^0,z)-F^T(x^0,z)\Vert \le ϵ.$$

$$\underset{(x_0,z)\in X\times Z}{\sup }|NE{T}_{N_i}(x_0,z)-F_i^T(x_0,z)|\le {\delta }_i.$$

$$S$$

$$m$$

$$\underset{(x,y)\in S}{\max }|xy-NET(x,y)|\le \frac{Y_{max}}{2^n},$$

$$S$$

$$m$$

$$\underset{(x,y)\in S}{\max }\left|\frac{x}{y}-NET(x,y)\right|\le \frac{1}{2^n},$$

$$0<Y_{\min }<y-{ϵ}_2,0\le {ϵ}_1\le x,0\le {ϵ}_2\le y,x/y\le Z_{\max }.$$

$$\left|\frac{x}{y}-\frac{x\pm {ϵ}_1}{y\pm {ϵ}_2}\right|\le \frac{Z_{\max }+1}{Y_{\min }}\max ({ϵ}_1,{ϵ}_2).$$

$$0\le x\le X_{\max },0\le y\le Y_{\max },$$

$$\max ({ϵ}_1,{ϵ}_2)\le \max (X_{\max },Y_{\max }),{ϵ}_1,{ϵ}_2\ge 0.$$

$$|xy-(x\pm {ϵ}_1)(y\pm {ϵ}_2)|$$

$$\le$$

$$\mathcal{H}:=\bigg{\{}h\mid H_{\min}\leq|h_{jk}|\leq H_{\max},\forall j,k,\;{\sum_{i=1}^{K}v(h)^{t}_{i}\geq{V_{\min}}},\forall t\bigg{\}}.$$

$$\displaystyle O\bigg{(}$$

$$\displaystyle O\bigg{(}T^{2}K^{2}\log\bigg{(}\max\bigg{(}K,P_{\max},H_{\max},\frac{1}{\sigma},\frac{1}{H_{\min}},\frac{1}{P_{\min}}\bigg{)}\bigg{)}+TK^{2}\log\bigg{(}\frac{1}{\epsilon}\bigg{)}\bigg{)},$$

$$\underset{h\in \mathcal{H}}{\max }\underset{i}{\max }|(v(h{)}_i^T{)}^2-NET(h,v^0{)}_i|\le ϵ$$

$$y=\min (\max (x,0),P_{\max }).$$

$$\left|\frac{x}{y}-\widetilde{\frac{x}{y}}\right|\le \frac{1}{2^n}.$$

$$m:=\lceil \log (V_{\max })\rceil ,$$

$$z_i=\text{mbox}Binary[(z^{(i)}-2^{i}y)\ge 0]$$

$$z^{(i)}=z^{(i+1)}-\max (2^{i+1}y+M(z_{i+1}-1),0),$$

$$\sum _{i=-n}^m{2}^i{z}_i.$$

Full text

In this work, we address both the thereotical and practical aspects of DNN-based algorithm approximation with applications to wireless resource management. We first pin down a class of optimization algorithms that are ‘learnable’ in theory by a fully connected DNN. Then, we focus on DNN-based approximation to a popular power allocation algorithm named WMMSE (Shi et al 2011). We show that using a DNN to approximate WMMSE can be fairly accurate – the approximation error $\epsilon$ depends mildly [in the order of $\log(1/\epsilon)$] on the numbers of neurons and layers of the DNN. On the implementation side, we use extensive numerical simulations to demonstrate that DNNs can achieve orders of magnitude speedup in computational time compared to state-of-the-art power allocation algorithms based on optimization.

I Introduction

Resource management tasks, such as transmit power control, transmit/receive beamformer design, and user admission control, are critical for future wireless networks. Extensive research has been done to develop various resource management schemes; see the recent overview articles [2, 3]. For decades, numerical optimization has played a central role in addressing wireless resource management problems. Well-known optimization-based algorithms for such purposes include those developed for power control (e.g., iterative water-filling type algorithms [4, 5], interference pricing [6], SCALE [7]), transmit/receive beamformer design (e.g., the WMMSE algorithm [8], pricing-based schemes [9], semidefinite relaxation based schemes [10]), admission control (e.g., the deflation based schemes [11], convex approximation based schemes [12]), user/base station (BS) clustering (e.g., the sparse optimization based schemes [13]), just to name a few. Simply speaking, these algorithms all belong to the class of iterative algorithms, which take a given set of real-time network parameters like channel realizations and signal to noise ratio specifications as their inputs, run a number of (potentially costly) iterations, and produce the “optimized” resource allocation strategies as their outputs.

Despite the fact that excellent performance of many of these algorithms has been observed through numerical simulations and theoretical analysis, implementing them in real systems still faces many serious obstacles. In particular, the high computational cost incurred by these algorithms has been one of the most challenging issues. For example, WMMSE-type algorithms require complex operations such as matrix inversion and bisection in each iteration [8, 14]. Water-filling type algorithms such as [15] and interference pricing algorithms [6] involve singular value decomposition at each iteration (when dealing with Multiple-Input-Multiple-Output (MIMO) interference systems). Algorithms such as the deflation-based joint power and admission control require successively solving a series of (possibly large-scale) linear programs. The computationally demanding nature of these algorithms makes real-time implementation challenging, because wireless resource management tasks are typically executed in a time frame of milliseconds (due to the fast changing system parameters such as channel conditions, number of users, etc.).

In this work, we propose a new machine learning-based approach for wireless resource management; see Figure 1 for an illustration of the system schematics. The main idea is to treat a given resource optimization algorithm as a “black box”, and try to learn its input/output relation by using a deep neural network (DNN) [16]. Our motivation is that if a network with several layers can well approximate a given resource management algorithm, then passing the algorithm input through the network to produce an output can be rather economical in computation – it only requires several layers of simple operations such as matrix-vector multiplications. Therefore, such an approximation, if accurate, can reduce processing time for wireless resource allocation substantially. In addition, training such a network is fairly convenient, since training samples are easily obtainable by running the optimization algorithm of interest on simulated data.

In the machine learning community, there have been several attempts of approximating an iterative optimization algorithm using DNNs in recent years. The work in [17] proposed to use a multilayer network to approximate the iterative soft-thresholding algorithm (ISTA) for sparse optimization [18]. In particular, the ISTA algorithm is first “unfolded” iteration by iteration as a chain of operations, and each of its iterations is mimicked by a layer of the network. Then the parameters of the network are learned by offline training. The authors of [19, 20] applied similar ideas for nonnegative matrix factorization and the alternating direction method of multipliers (ADMM), respectively. Recently, the authors of [21] proposed to learn the per-iteration behavior of gradient based algorithms, and the authors of [22] proposed to use reinforcement learning for the same task. In a nutshell, these algorithms are more or less based on the same “unfolding” idea, which is feasible either because the iterations have simple structure such as in ISTA or due to the fact that gradient information is assumed to be known like in [21] and [22]. However, for complex algorithms, it is no longer clear whether an iteration can be explicitly modeled using a structured layer of a neural network. Further, it is important to note that none of these existing works have provided rigorous theory to back their approaches – e.g., it is unclear whether or not algorithms such as ISTA can be accurately approximated using the unfolding idea.

It is worth noting that there have been a number of works that apply DNN for various communication tasks. For example, the recent works in [23, 24, 25, 26] have demonstrated promising performance of using DNN in a number of tasks such as anomaly detection and decoding. However, the focus in these papers is signal modeling rather than algorithm approximation. The very recent work in [27] proposes a DNN architecture to approximate an algorithm for the MIMO detection problem, but the idea, again, follows the unfolding work in [17].

Our approach is very different from the unfolding idea: We propose to employ a generic fully connected DNN to approximate a wireless resource allocation algorithm, rather than unfolding the algorithms of interest. Unlike the algorithms with relatively simple structure that could be easily unfolded, wireless resource management algorithms often entail computationally heavy iterations involving operations such as matrix inversion, SVD, and/or bi-section. Therefore, their iterations are not amenable to approximation by a single layer of the network.

Overall, the DNN-based algorithm approximation is an emerging and largely unexplored topic. Therefore its success will be critically dependent on the answer to the following challenging research questions: What kind of optimization algorithms are “learnable”, meaning they can be approximated arbitrarily well by a neural network? How many layers / neurons are needed to approximate a resource allocation algorithm accurately? How robust or sensitive is the method in practice?

Contributions. The main contribution of this work is three-fold: First, we propose the first deep learning based scheme for real-time resource management over interference-limited wireless networks, which bridges the seemingly unrelated areas of machine learning and wireless resource allocation (in particular, power control over interference networks). Second, unlike all existing works on approximating optimization algorithms such as those using unfolding, our approach is justified by rigorous theoretical analysis. We show that there are conditions under which an algorithm is leanrable by a DNN, and that the approximation error rate is closely related to the ‘depth’ and ‘width’ of the employed DNN. These theoretical results provide fundamental understanding to the proposed “learn to optimize” approach. They also offer insights and design guidelines for many other tasks beyond wireless resource allocation. Third, extensive numerical experiments using simulated data and real DSL data are conducted to demonstrate the achievable performance of the proposed approach under a variety of scenarios. As a proof-of-concept, the results provided in this paper indicate that DNNs have great potential in the real-time management of the wireless resource.

To promote reproducible research, the codes for generating most of the results in the paper are made available on the authors’ website: https://github.com/Haoran-S/TSP-DNN.

Notation. We use $I_{X}(x)$ to denote the indicator function for a set $X$; the function takes value $1$ if $x\in X$, and its takes value [math] if $x\notin X$; We use $\mathcal{N}(m,\sigma^{2})$ to denote normal distribution with mean $m$ and variance $\sigma^{2}$. A Rectified Linear Unite (ReLU) is defined as $\mbox{ReLU}(x)=\max\{0,x\}$; A binary unit, denoted as $\mbox{Binary}(x)$, returns $1$ if $x$ evaluates as true, and returns [math] otherwise. We use $\mathbb{R}_{+}$ to denote the set of all nonnegative numbers.

II Preliminaries

II-A System Model

We consider the following basic interference channel (IC) power control problem, for a wireless network consisting of $K$ single-antenna transceivers pairs. Let $h_{kk}\in\mathbb{C}$ denote the direct channel between transmitter $k$ and receiver $k$, and $h_{kj}\in\mathbb{C}$ denote the interference channel from transmitter $j$ to receiver $k$. All channels are assumed to be constant in each resource allocation slot. Furthermore, we assume that the transmitted symbol of transmitter $k$ is a Gaussian random variable with zero mean and variance $p_{k}$ (which is also referred to as the transmission power of transmitter $k$). Further, suppose that the symbols from different transmitters are independent of each other. Then the signal to interference-plus-noise ratio (SINR) for each receiver $k$ is given by

[TABLE]

where $\sigma_{k}^{2}$ denotes the noise power at receiver $k$.

We are interested in power allocation for each transmitter so that the weighted system throughput is maximized. Mathematically, the problem can be formulated as the following nonconvex problem

[TABLE]

where $P_{\max}$ denotes the power budget of each transmitter; $\{\alpha_{k}>0\}$ are the weights. Problem (1) is known to be NP-hard [28]. Various power control algorithms have been proposed [8, 6, 7], among which the WMMSE algorithm has been very popular. In what follows, we introduce a particular version of the WMMSE applied to solve the power control problem (1).

II-B The WMMSE Algorithm

The WMMSE algorithm converts the weighted sum-rate (WSR) maximization problem to a higher dimensional space where it is easily solvable, using the well-known MMSE-SINR equality[29], i.e., $\mbox{mmse}_{k}=\nicefrac{{1}}{{(1+\mbox{sinr}_{k})}}$, where “$\mbox{mmse}_{k}$” denotes the minimum-mean-squared-error (MMSE) of user $k$.

Note that the original WMMSE [8] is designed for problems whose design variables are beamformer vectors with complex entries. In this paper, in order to simplify our subsequent implementation of DNN based schemes, we modify the algorithm so that it can work in the real domain. Specifically, we first observe that the rate function remains unchanged if $h_{kj}$ is replaced by $|h_{kj}|$. Using this observation, one can show (by following the steps in [8, Theorem 1]) that problem (1) is equivalent to the following weighted MSE minimization problem

[TABLE]

where the optimization variables are all real numbers, and we have defined

[TABLE]

Here by equivalent we meant that all stationary solutions of these two problems are identical to each other.

The WMMSE solves (2) using the block coordinate descent method [30], i.e., each time optimizing one set of variables while keeping the rest fixed; see Fig. 2 for its detailed steps. It has been shown in [8] that the WMMSE is capable of reaching a stationary solution of problem (1). Note that since the interfering multiple-access channel (IMAC) can be viewed as a special IC with co-located receivers, the WMMSE algorithm in Fig. 1 can be applied to solving the power allocation problem of IMAC as well 111In the IMAC, we assume that within each cell the BS does not perform successive interference cancellation..

III The Proposed Approach

In this section, we present the proposed approach that uses DNN to approximate WMMSE. We first present two theoretical results to justify our approach, followed by a detailed description of how the DNN is set up, and how training, validation and testing stages are carried out.

III-A Universal Approximation

At this point, it remains unclear whether a multi-layer neural network can be used to approximate the behavior of a given iterative algorithm, like WMMSE, for solving the nonconvex optimization problem (1). The answer to such a question turns out to be nontrivial. To see where the challenge is, let us consider the following example.

Example. We consider approximating the behavior of the classic gradient descent (GD) for solving the following problem

[TABLE]

where $x$ is the optimization variable and $z$ is the problem parameter. The GD iteration is given by the following

[TABLE]

where $t$ is the iteration number; $x^{0}$ is the randomly generated initial solution; $\alpha>0$ is the stepsize; $T$ is the total number of iteration.

Given a set of training data points $\{z^{(i)},(x^{T})^{(i)}\}$, we use a simple three layer neural network to approximate the relationship $z\to x^{T}$, which characterizes the behavior of GD. Unfortunately, the network learns a model that only outputs zero, regardless what the inputs are; see Figure 3 (a).

The reason that we are not learning the correct behavior of GD is that, problem (4) is a nonconvex problem with multiple isolated stationary solutions $\pm\sqrt{z}$ when $z>0$. Depending on what the initial solution $x^{0}$ is, the GD can converge to either a positive solution or the negative one. This behavior confuses the neural network, so that only the middle point is learned, which turns out to be a non-interesting solution.

To resolve such an issue, we consider learning a new relationship $(x^{0},z)\to x^{T}$ by using the same three layer network. Interestingly, now the behavior of GD can be accurately approximated, and the results are shown in Figure 3 (b). $\blacksquare$

The above example indicates that a rigorous theoretical analysis is needed to characterize how we should use neural networks to learn optimization algorithms. To begin our analysis, let $NET_{N}(x^{0},z)$ denote the output of a feedforward network, which has one hidden layer with $N$ sigmoid activation functions, and has $(x^{0},z)$ as input. Then consider a finite step iterative algorithm, in which the relationship between $z^{t}$ and $z^{t+1}$ (i.e., its $t^{\rm th}$ and ${t+1}^{\rm th}$ iterates) is given by

[TABLE]

where $f^{t}$ is a continuous mapping representing the $t^{\rm th}$ iteration; $z\in Z$ is the problem parameter; $x^{t},x^{t+1}\in X$ where $Z$ and $X$ represent the parameter space and the feasible region of the problem, respectively. We have the following result as an extension of the well-known universal approximation theorem for multilayer feedforward networks [31].

Definition 1

Denote $i^{th}$ coordinate of $y$ as $y_{i}$, a function $y=f(x),x\in R^{m},y\in R^{n}$ is a continuous mapping when $y_{i}=f_{i}(x)$ is a continuous function for all $i$.

Theorem 1

(Universal approximation theorem for iterative algorithm) Suppose that $Z$ and $X$ are certain compact sets. Then the mapping from problem parameter $z$ and initialization $x^{0}$ to final output $x^{T}$, i.e.,

[TABLE]

can be accurately approximated by $NET_{N}(x^{0},z)$ (which represents an N hidden units network with sigmoid activation function) in the following sense: For any given error $\epsilon>0$, there exists a positive constant $N$ large enough such that

[TABLE]

Proof. The proof is relatively straightforward. Since every $f^{t}$ is a continuous mapping and composition of continuous mappings is still a continuous mapping, we have for any $i$, the $i^{\rm th}$ coordinate of $x^{T}$ is a continuous function of $(x^{0},z)$, denote by $x^{T}_{i}=F^{T}_{i}(x^{0},z)$. From [31, Theorem 2.1], for any compact set $X\times Z$ and any $\delta_{i}>0$, there exist $N_{i}$ such that

[TABLE]

Suppose $x^{T}$ has dimension $L$. Then by stacking $L$ of the above networks together, we can construct a larger network $\hat{x}=NET_{N}(x_{0},z)$ with $N=\sum_{i}^{L}N_{i}$. Given $\epsilon$, we can choose $\delta_{i}=\frac{\epsilon}{\sqrt{L}}$, then we obtain (6). $\blacksquare$

Remark 1

Our result reveals that for any deterministic algorithm whose iterations represent continuous mappings, we can include its initialization as an additional input feature, and learn the algorithm behavior by a well trained neural network. If the underlying optimization problem is nonconvex [as the WSR maximization problem (1)], having the initialization as an input feature is necessary, since without it the mapping $(z)\to x^{T}$ is not well defined (due to the possibility of the algorithm converging to multiple isolated solutions). Alternatively, it is also possible to learn the mapping $z\to x^{T}$, for a fixed initialization $x^{0}$.

Remark 2

It can be verified that each iteration of WMMSE represents a continuous mapping, and that the optimization variable lies in a compact set. Therefore, by assuming that the set of channel realizations $\{h_{ij}\}$ lies in a compact set, using Theorem 1 we can conclude that WMMSE can be approximated arbitrarily well by a feedforward network with a single hidden layer. $\blacksquare$

III-B Approximation Rate of WMMSE via DNN

Our theoretical result in the previous section indicates that a large class of algorithms, including WMMSE, can be approximated very well by a neural network with three layers. However, such a network may not be implementable because it is not clear whether a finite-size network is able to achieve a high quality approximation. Therefore, a more relevant and challenging question is: can we rigorously characterize the network size (i.e., the number of layers and total number of neurons) required to achieve a given approximation error?

In this section, we show that the answer is affirmative. We will use the WMMSE algorithm as an example, and provide sharper approximation guarantees than what has been obtained in Theorem 1. In particular, we show that under certain network structure, the size of the network scales with the approximation error $\epsilon$ in the rate of $\mathcal{O}(\log(1/\epsilon))$.

The main steps of the analysis is as follows: S1) Construct simple neural networks that consist of ReLUs and Binary Units to approximate multiplication and division operations; S2) Compose these small neural networks to approximate a rational function representing one iteration of the algorithm; S3) Concatenate these rational functions to approximate the entire algorithm; S4) Bounding the error propagated from the first iteration to the last one. All the proofs in this section are delegated to the Appendix.

Our first two lemmas construct neural networks that can approximate multiplication and division operations.

Lemma 1

For any two positive numbers $X_{\max},Y_{\max}$, define

[TABLE]

There exists a multi-layer neural network with input $(x,y)$ and output $NET(x,y)$ satisfying the following relation

[TABLE]

where the network has $O(m+n)$ layers and $O(m+n)$ binary units and ReLUs.

Lemma 2

For any positive number $Y_{\max}$ $Z_{\max}$, define

[TABLE]

There exists a neural network with input $(x,y)$ and ouput $NET(x,y)$ satisfying the following relation

[TABLE]

where the network has $O(m+n)$ layers and $O(m+n)$ binary units and ReLUs.

We note that in [32] the authors have shown that the function $z=x^{2}$ can be approximated by a neural network with ReLU units and binary units. Our construction of $xy$ follows that in [32], while the construction of the division operation $x/y$ appears to be new.

At this point, it appears that we can concatenate a series of ‘division’ and ‘multiplication’ networks to approximate any rational function. However, care has to be exercised here since when performing composition, the approximation errors from the inner functions will propagate to the outer ones. It is essential to control these errors in order to obtain the overall approximation bounds. Below we provide analysis to characterize such an error propagation process.

Lemma 3

Given positive numbers $Y_{\min}$, $Z_{\max}$, $\epsilon_{1}$ and $\epsilon_{2}$. Suppose that $(x,y)$ satisfies the following relations

[TABLE]

Then we have

[TABLE]

Lemma 4

Suppose positive numbers $X_{\max}$, $Y_{\max}$, $\epsilon_{1}$ and $\epsilon_{2}$ satisfy the following relations

[TABLE]

Then we have

[TABLE]

Using the above four lemmas, we can obtain our main theoretical result in this section. To concisely state the result, let us make the following definitions. Given an input channel vector $h:=\{h_{ij}\}\in\mathbb{R}^{K^{2}}$, let us use $v(h)^{t}_{i}$ to denote the variable $v_{i}$ at $t^{\rm th}$ iteration generated by WMMSE. Also let $H_{\min},H_{\max}>0$ denote the minimum and maximum channel strength, and let $V_{\min}>0$ be a given positive number.

Theorem 2

Suppose that WMMSE is randomly initialized with $(v^{0}_{k})^{2}\leq P_{\max},\ \sum_{i=1}^{K}v(h)^{0}_{i}\geq V_{\min},$ and it is executed for $T$ iterations. Define the following set of ‘admissible’ channel realizations

[TABLE]

Given $\epsilon>0$, there exists a neural network with $h\in\mathbb{R}^{K^{2}}$ and $v^{0}\in\mathbb{R}_{+}^{K}$ as input and $NET(h,v^{0})\in\mathbb{R}_{+}^{K}$ as output, with the following number of layers

[TABLE]

and the following number of ReLUs and binary units

[TABLE]

such that the relation below holds true

[TABLE]

Corollary 1

The above statement still holds if the WMMSE algorithm has a fixed initialization with $(v^{0}_{k})^{2}=P_{\max}$, and the neural network has $h$ as input.

Remark 3

The admissible set $\mathcal{H}$ is defined mainly to ensure that the set of channels lie in a compact set. Admittedly, the conditions that the channel realizations are lower bounded by $H_{\min}$, and that they generate WMMSE sequences that satisfy $\sum_{i=1}^{K}v(h)^{t}_{i}\geq V_{\min}$ are somewhat artificial, but they are needed to bound the error propagation. It is worth noting that for problem (2), $v_{i}=0,\;i=1,\cdots,K$ is an isolated local maximum solution (i.e., the only local maximum solution within a small neighborhood). Therefore it can be proved that as long as the WMMSE algorithm (which belongs to the coordinate descent type method) is not initialized at $v^{0}_{i}=0,\;i=1,\cdots,K$, then with probability one it will not converge to it; see recent result in [33]. This fact helps justify the assumption that there exists $V_{\min}>0$ such that $\sum_{i=1}^{K}v(h)^{t}_{i}\geq V_{\min}$ for all $t$.

Remark 4

It is worth mentioning that, other operations such as binary search, and finding the solution for a linear systems of equation: $Ax=b$, can also be approximated using similar techniques. Therefore, many resource allocation algorithms other than WMMSE can be approximated following similar analysis steps as in Theorem 2, as long as they can be expressed as the composition of these ‘basic’ operations such as multiplication, division, binary search, threshloding operations, etc.

Remark 5

The bounds in Theorem 2 provide an intuitive understanding of how the size of the network should be dependent on various system parameters. A key observation is that having a neural network with multiple layers is essential in achieving our rate bounds. Another observation is that the effect of the approximation error on the size of the network is rather minor [the dependency is in the order of $\mathcal{O}(\log(1/\epsilon))$]. However, we do want to point out that the numbers predicted by Theorem 2 represent some upper bounds on the size of the network. In practice much smaller networks are often used to achieve the best tradeoff between computational speed and solution accuracy.

IV System Setup

In this section, we describe in detail the DNN architectures as well as how the training and testing stages are performed.

Network Structure. Our proposed approach uses a fully connected neural network with one input layer, multiple hidden layers, and one output layer as shown in Fig. 4. The input of the network is the magnitude of the channel coefficients $\{|h_{kj}|\}$, and the output of the network is the power allocation $\{p_{k}\}$. Further, we use ReLU as the activation function for the hidden layers activation. Additionally, to enforce the power constraint in (1) at the output of DNN, we also choose a special activation function for the output layer, given below

[TABLE]

Data Generation. The data is generated in the following manner. First, the channel realizations $\{h^{(i)}_{kj}\}$ are generated following certain distributions (to be specified in Section V), where the superscript $(i)$ is used to denote the index of the training sample. For simplicity we fix $P_{\max}$ and $\sigma_{k}$ for all $k$. Then, for each tuple $(P_{\max},\{\sigma_{k}\},\{|h^{(i)}_{kj}|\})$, we generate the corresponding optimized power vectors $\{p^{(i)}_{k}\}$ by running WMMSE, with $v^{0}_{k}=\sqrt{P_{\max}},\;\forall~{}k$ as initialization, and with $\mbox{obj}_{\rm new}-\mbox{obj}_{\rm old}<10^{-5}$ or $\mbox{number of iterations}>500$ as termination criteria. The tuple $(\{|h^{(i)}_{kj}|\},\{p^{(i)}_{k}\})$ is referred to as the $i$th training sample. Then we repeat the above process for multiple times to generate the entire training data set, as well as the validation data set. The validation set is used to perform the cross-validation, model selection and early stopping during the training stage. Typically, the size of the validation data set is small compared with that of the training set. We use $\mathcal{T}$ and $\mathcal{V}$ to collect the indices for the training and validation sets, respectively.

Training Stage. We use the entire training data set $(\{|h^{(i)}_{kj}|\},\{p^{(i)}_{k}\})_{i\in\mathcal{T}}$ to optimize the weights of the neural network. The cost function we use is the mean squared error between the label $\{p^{(i)}_{k}\}$ and the output of the network. The optimization algorithm we use is an efficient implementation of mini-batch stochastic gradient descent called the RMSprop algorithm, which divides the gradient by a running average of its recent magnitude [34]. We choose the decay rate to be 0.9 as suggested in [34] and select the proper learning rate and batch size by cross-validation. To further improve the training performance, we initialize the weights using the truncated normal distribution 222We generate a variable from the truncated normal distribution in the following manner: First, generate a variable from standard normal distribution. Then if its absolute value is larger than 2, it is dropped and re-generated.. Furthermore, we divide the weights of each neuron by the square root of its number of inputs to normalize the variance of each neuron’s output[35].

Testing Stage. In the testing stage, we first generate the channels following the same distribution as the training stage. For each channel realization, we pass it through the trained network and collect the optimized power. Then we compute the resulting sum-rate of the power allocation generated by DNN and compare it with that obtained by the WMMSE. We also test the robustness of the learned models by generating channels following distributions that are different from the training stage, and evaluate the resulting performance.

V Numerical Results

V-A Simulation Setup

The proposed DNN approach is implemented in Python 3.6.0 with TensorFlow 1.0.0 on one computer node with two 8-core Intel Haswell processors, two Nvidia K20 Graphical Processing Units (GPUs), and 128 GB of memory. The GPUs are used in the training stage to reduce the training time, but are not used in the testing stage. To rigorously compare the computational performance of WMMSE with the proposed approach, we have implemented the WMMSE in both Python and C. The Python implementation is used to compare the computational performance under the same platform while the C implementation is used to achieve the best computational efficiency of WMMSE.

We consider the following two different channel models:

Model 1: Gaussian IC. Each channel coefficient is generated according to a standard normal distribution, i.e., Rayleigh fading distribution with zero mean and unit variance. Rayleigh fading is a reasonable channel model that has been widely used to simulate the performance of various resource allocation algorithms. In our experiment, we consider three different network setups with $K\in\{10,20,30\}$.

Model 2: IMAC. For practical consideration, a multi-cell interfering MAC (IMAC) model is considered with a total of $N$ cells and $K$ users. The cell radius $R$ (i.e., the half distance between centers of adjacent cells) is set to be $100$ meters. In each cell, one BS is placed at the center of the cell and the users are randomly and uniformly distributed in the area from the inner circle of radius $r$ to the cell boundary; see Fig. 5 for an illustration of the network configuration. We assume that the cells all have the same number of users. The channel between each user and each BS is randomly generated according to a Rayleigh fading distribution with zero mean and variance $(200/d)^{3}L$, where $d$ denotes the distance between the BS and user, and $\mbox{10log10}(L)$ following a normal distribution with zero mean and variance $64$; see [36] for a similar network setup. In our experiment, we consider five difference network scenarios, with $(N,K)\in\{(3,12),(3,18),(3,24),(7,28),(20,80)\}$.

We mention that for each network scenario (i.e., IC/IMAC with a different number of BSs/users), we randomly generate one million realizations of the channel as the training data and ten thousand realizations of the channel as the validation data and testing data. We use the distinctive random seeds for training, validation and testing data.

V-B Parameter Selection

We choose the following parameters for the neural network. For all our numerical results, we use a network with three hidden layers, one input layer, and one output layer. Each hidden layer contains $200$ neurons. The input to the network is the set of channel coefficients, therefore the size of it depends on the channel models. More specifically, for Gaussian IC the input size is $K^{2}$ while for the IMAC it is $N\times K$. The output is the set of power allocations, therefore its size is $K$ for both Gaussian IC and IMAC.

To find parameters for the training algorithm, we perform cross-validation for different channel models as follows:

Model 1: Gaussian IC. We study the impact of the batch size and learning rate on the MSE evaluated on the validation set, as well as the total training time. Based on the result shown in Fig. 6, we choose the batch size to be $1000$ and the initial learning rate to be $0.001$. Then we choose to gradually decrease the learning rate when the validation error does not decrease.

Model 2: IMAC. The parameter selection process is the same as that in the Gaussian IC channel.

V-C Sum-Rate Performance

We evaluate the sum-rate performance of the DNN-based approach in the testing stage compared to the following schemes: 1) the WMMSE; 2) the random power allocation strategy, which simply generates the power allocation as: $p_{k}\sim\mbox{Uniform}(0,P_{\max})$, $\forall~{}k$; 3) the maximum power allocation: $p_{k}=P_{\max}$, $\forall~{}k$; The latter two schemes serve as heuristic baselines. The simulation results are shown in Fig. 7 for both the Gaussian IC and the IMAC. Each curve in the figure represents the result obtained by averaging over 10, 000 randomly generated testing data points. It is observed that the sum-rate performance of DNN is very close to that of the WMMSE, while significantly outperforming the other two baselines. It is worth noting that for both Gaussian IC and IMAC, we have observed that the “optimized” allocation strategies obtained by WMMSE are binary in most cases (i.e., $p_{k}$ takes the value of either [math] or $P_{\max}$). Therefore, we also discretize the prediction to binary variables to increase the accuracy. Specifically, for each output $y$ of our output layer, we round the prediction up to one if $y>0.5$, and round down it to zero if $y<0.5$.

V-D Scalability Performance

In this subsection, we demonstrate the scalability of the proposed DNN approach when the size of the wireless network is increased. The average achieved sum-rate performance (averaged using $10,000$ test samples) and the percentage of achieved sum-rate of DNN over that of the WMMSE are presented in TABLE I (for the IC) and TABLE II (for the IMAC). It can be seen that our proposed method achieves good scalability for prediction accuracy and computational efficiency. For example, compared with the IMAC WMMSE (which is implemented in C) with $N=20$ and $K=80$, the DNN obtains $90.36\%$ sum-rate while achieving more than one hundred times (two orders of magnitude) speed up. Additionally, in Fig. 8, we show the distribution of the sum-rates over the entire test dataset. It is observed that the DNN approach gives a fairly good approximation of the entire rate profile generated by the WMMSE.

Moreover, from Table I and Table II, we observe that the DNN performs better in the IMAC setting, achieving lower computational time and higher relative sum-rate (relative to WMMSE). The improved relative computational performance in IMAC is due to the fact that, for the same number of users $K$, the IMAC has $K\times N$ different input channel coefficients while the IC has $K^{2}$. Therefore when $N\ll K$, the DNN built for IMAC has a smaller size compared with that of the IC. On the contrary, the computation steps for WMMSE are the same for IMAC and IC, where at each iteration steps 7-9 in Fig. 2 are executed for each user $K$. These facts combined together result in better real-time computational efficiency for DNN in the IMAC setting. On the other hand, the improved relative sum rate performance in IMAC is mainly due to the fact that the IMAC channels are more ‘structured’ (which are generated according to actual users locations and shadow fading) compared with the IC channels (which are simply randomly generated from Gaussian distribution). This means that the training and testing samples for the IMAC are closely related to each other, leading to better generalization capabilities of the DNN model in the testing stage.

V-E Generalization Performance

In the previous subsection, we have demonstrated the superiority of our DNN approach, when the distribution of the testing data is the same as that of the training data. However, in practice, there could be significant model mismatch between the training and the testing dataset. For example, the number of users in the system is constantly changing, but it is impossible to train a different neural network for each network configuration. An interesting question is: Whether the model learned by the DNN is able to adapt to such training-testing model mismatch? In this subsection, we investigate the generalization performance of the proposed DNN approach.

Model 1: Gaussian IC. We study the scenario in which only $K/2$ users are present in the testing, while the DNN is trained with $K$ users (we name this case “half user” in TABLE III). The purpose is to understand the generalization capability of the trained model when input dimension is different. Somewhat surprisingly, we observe that in this case, the DNN still performs well. This result suggests that it is not necessary to train one DNN for each network configuration, which makes the proposed approach very attractive for practical use. Note that in TABLE III the input size for WMMSE is reduced from $K^{2}$ to $K^{2}/4$ while the input size for DNN is still $K^{2}$ (the unused inputs are replaced by zeros). Therefore compared with the results in TABLE I, we can observe that the computational time for WMMSE has been significantly reduced.

Model 2: IMAC. For the IMAC model with fixed number of base stations $N$, there could be many choices of network parameters, such as $R,r$ and $K$. We study the cases that the DNN is trained with $R=100$ meters and $r=0$ meters, and apply the resulting model to networks with with $R$ ranging from 100 to 500 meters, and $r$ from 0 to 99 meters (note that $r=99$ meters and $R=100$ meters represents the challenging case where all the users are on the cell boundary). The results are shown in TABLE IV and TABLE V, from which we can conclude that our model generalizes relatively well when the testing network configurations are sufficiently close to those used in the training. Further, as expected, when the testing models are very different, e.g., $R=500$ meters, the performance degrades.

V-F Real Data Performance

In this section, we further validate our approach using real data. We consider the interference management problem over the very-high-bit-rate digital subscriber line (VDSL) channels, which can be cast as a power control problem over an interference channel [2]. The data is collected by France Telecom R$\&$D; see [37, 38] and the references therein. The measured lengths of the VLSI lines were 75 meters, 150 meters, and 300 meters, each containing $28$ loops, and the bandwidth is up to 30 MHz. For each length, all 378 (28 choose 2) crosstalk channels of the far-end crosstalk (FEXT) and near-end crosstalk (NEXT) were measured, for a total of over 3000 crosstalk channels. There is a total of $6955$ channel measurements[37, 38].

Problem setup. We model the VDSL channel using a $28$-user IC, and we use the magnitude of each channel coefficient to compute the power allocation (by using WMMSE). The entire 6955 channel samples are divided into 5000 samples of validation set and 1955 samples of testing set. The training data set is computer-generated following channel statistics learned from the validation set. Specifically, we first calculate the means and variances for both the direct channel and the interfering channels in the validation data set, denoted as $(m_{d},\sigma^{2}_{d})$, $(m_{i},\sigma^{2}_{i})$, respectively. Then we randomly generate 28 direct channel coefficients independently using $\mathcal{N}(m_{d},\sigma^{2}_{d})$, and $28\times 27$ interfering channels from $\mathcal{N}(m_{i},\sigma^{2}_{i})$ to form one training data sample. Repeating the above process, we generate $50,000$ training samples for each measured length.

Using the training, validation and testing data set as described above, we perform the training and testing following the procedures outlined in Section IV. Since the channel coefficients are relatively small compared to our previous random generated channel, we set the environment noise variance to $0.001$ to make sure that WMMSE gives meaningful output.

Numerical results. The results are shown in TABLE VI. Despite the fact that the total available data set is quite limited, the proposed DNN approach can still achieve relatively high sum-rate performance on the measured testing data set. This result is encouraging, and it further validates our ‘learning to optimize’ approach.

VI Perspectives and Future Work

In this work, we design a new ‘learning to optimize’ based framework for optimizing wireless resources. Our theoretical results indicate that it is possible to learn a well-defined optimization algorithm very well by using finite-sized deep neural networks. Our empirical results show that, for the power control problems over either the IC or the IMAC channel, deep neural networks can be trained to well-approximate the behavior of the state-of-the-art algorithm WMMSE. In many aspects, our results are encouraging. The key message: DNNs have great potential as computationally cheap surrogates of expensive optimization algorithms for quasi-optimal and real-time wireless resource allocation. We expect that they could also be used in many other computationally expensive signal processing tasks with stringent real-time requirements. However, the current work still represents a preliminary step towards understanding the capability of DNNs (or related learning algorithms) for this type of problems. There are many interesting questions to be addressed in the future, and some of them are listed below:

How to further reduce the computational complexity of DNNs? 2. 2.

Is it possible to design specialized neural network architectures (beyond the fully connected architecture used in the present work) to further improve the real-time computation and testing performance? 3. 3.

Is it possible to generalize our proposed approach to more challenging problems such as beamforming for IC/IMAC?

Appendix A Proofs

Proof. [Proof of Lemma 2] Let us denote the approximated ${x}/{y}$ as $\widetilde{(x/y)}$, with $(x,y)\in{S}$. The idea is to approximate ${x}/{y}$ by its finite binary expansison $\widetilde{(x/y)}=\sum_{i=-n}^{m}{2^{i}z_{i}}$, where $z_{i}$ is the $i$-th digit of binary expansion of ${x}/{y}$. The approximation error will be bounded by

[TABLE]

To proceed, let us define

[TABLE]

where ‘$\log$’ is logarithm with base 2. The $i$-th digit of binary expansion of ${x}/{y}$ (the bit corresponding to $2^{i}$) can be written as

[TABLE]

where $z^{(i)}$ is defined below recursively

[TABLE]

with $M\geq 2^{m}Y_{\max}$ being a large constant and $z^{(m)}=x$. Note that $z^{(i)}$ can be interpreted as ${x}/{y}\ \mbox{mod}\ 2^{i+1}$.

From the above construction, it is clear that we need two layers to represent each $z_{i}$, where one ReLU in the first layer is used to represent $z^{(i)}$, and one Binary unit in second layer is used to represent $z_{i}$. An illustration can be found in Fig. 9.

Overall, we need $2(m+n+1)$ layers and $(m+n+1)$ Binary units and $(m+n)$ ReLUs to represent all $z_{i}$’s. Once all $z_{i}$’s are constructed, $\widetilde{(x/y)}$ is given by

[TABLE]

This completes the proof.

Proof. [Proof of Lemma 1] The polynomial $xy$ can be approximated by $\tilde{x}y$ where $\tilde{x}$ is the $n$-bit binary expansion of $x$.

[TABLE]

where $x_{i}$ is $i$-th bit of binary expansion of $x$ and we assume $y\leq Y_{\max},\forall y$.

Clearly, the approximation error will be at most

[TABLE]

since $(x-\tilde{x})\leq\frac{1}{2^{n}}$.

The $i$-th digit of binary expansion of $x$ (the bit corresponding to $2^{i}$) can be done by substituting $y=1$ into $x/y$ in the proof of Lemma 2, leading to

[TABLE]

where $x^{(i)}$ have following recurrent definition

[TABLE]

and $x^{(m)}=x$, $x^{(i)}$ can be interpreted as $x\ mod\ 2^{i+1}$.

Obviously, the number of layers and nodes for multiplication is similar to which in the proof of Lemma 2, the only difference is that one more layer and $(m+n+1)$ ReLUs are added to achieve equation (19).

Proof. [Proof of Lemma 3] Given the following

[TABLE]

we have

[TABLE]

This completes the proof.

Proof. [Proof of Lemma 4] Given $0\leq x\leq X_{max},$ $0\leq y\leq Y_{max},$ $\max(\epsilon_{1},\epsilon_{2})\leq\max(X_{\max},Y_{\max}),\;\epsilon_{1},\epsilon_{2}\geq 0,$ we have

[TABLE]

This completes the proof.

Proof. [Proof of Theorem 2] The idea is to approximate $v_{k}^{t},a_{k}^{t},b_{k}^{t}$ [defined by (22) - (24)], $k={1,2,...,K}$ using a neural network with inputs $v_{k}^{t-1},a_{k}^{t-1},b_{k}^{t-1},k={1,2,...,K}$ and $\{|h_{jk}|\}$. By repeating the network $T$ times and concatenating them together, we can approximate $T$ iterations of WMMSE.

Let us define $\tilde{v}^{t}_{k}$ to be the output of the neural network that approximates the $t$th iteration of WMMSE. We will begin upper-bounding the error

[TABLE]

and derive the relationship between upper bound of approximation error and network size.

To proceed, note that the update rules of WMMSE algorithm can be rewritten in the following form

[TABLE]

We will approximate each iteration of WMMSE algorithm by a deep neural network and repeat the network $T$ times to approximate $T$ iterations of WMMSE. By using inequalities (10) and (13), we can derive the upper bound of approximation error after $T$ iterations.

Firstly, we assume

[TABLE]

Furthermore, we assume for any $h\in\mathcal{H}$ and $t=0,1,2,...,T$, $\sum_{j=1}^{K}v_{k}^{t}\geq P_{\min}$.

We note that $a_{k}^{t}$ is a rational function of $h_{kj}$ and $v_{j}^{t},j=1,2,...,K$, and we can approximate any rational function by repeatedly applying approximation to multiplication and division operations. We can construct an approximated version of $a_{k}^{t}$ by the following

[TABLE]

where $\widetilde{xy}$ means we will approximate $xy$ using a neural network by leveraging Lemma 1. From (25) and Lemma 1, the error in numerator of (26) is upper-bounded by

[TABLE]

where the first equality is from $n$-bit binary expansion of $\tilde{v}_{k}^{t}$ in $\widetilde{|h_{kk}|^{2}\tilde{v}_{k}^{t}}$ and $\zeta<\frac{1}{2^{n}}$ is error induced by this binary expansion.

Similarly, for denominator of (26), we have

[TABLE]

and

[TABLE]

Then, by Lemma 3 and Lemma 2, the approximation error of (26) is upper bounded by

[TABLE]

Using the same method, we can approximate $b_{k}^{t}$ by

[TABLE]

And the error is upper bounded by

[TABLE]

The last result is obtained by using Lemma 1 - 4.

Finally, we approximate $v_{k}^{t+1}$ as

[TABLE]

By using Lemma 1 - 4 again, the approximation error of (28) can be upper-bounded by the following

[TABLE]

where the constant $G$ is given below

[TABLE]

Since we use the big $O$ notation in the theorem, in derivation of $G$, we assumed $H_{\max}\geq 1$ and $P_{\max}\geq 1$ so that the these two number will dominate constant 1 when propagating error, i.e. $H_{\max}\theta+\theta\leq 2H_{\max}\theta$ for any positive number $\theta$.

Suppose we use $n$-bit binary expansion for all variables and $\epsilon\geq 2^{-n}$, then (29) reduces to

[TABLE]

where the last inequality is due to (25). Thus, the approximation error is amplified at most $G$ times through approximation of one iteration.

Since $G\geq 1$, if we first approximate the initialization $v_{k}^{0}$ with $n$-bit binary expansion, the approximation error upper bound of latter iterations will be grater than $2^{-n}$, then using the method above to approximate $T$ iterations of WMMSE, the resulting error will be upper bounded by

[TABLE]

which means

[TABLE]

By counting the number of neurons and layers used for approximation, we can derive the trade-off between the approximation error upper bound and the number of units (RelU and binary) and number of layers we use.

To approximate $a_{k}^{t}$, we need $\lceil\log(P_{\max})/2\rceil+n+1$ bits for each $\tilde{v_{k}^{t}}$ (since $v_{k}^{t}\leq\sqrt{P_{max}}$), $\lceil\log(P_{\max})\rceil+n+1$ bits for each $\widetilde{\tilde{v_{k}}^{t}\tilde{v_{k}}^{t}}$, and $\lceil 2\log(H_{\max})+\log(P_{\max})/2+2\log{\frac{1}{\sigma_{k}}}\rceil+n+1$ bits for the division operation. Over all, we need at most

[TABLE]

bits for each $a_{k}^{t}$.

For approximating $b_{k}^{t}$, we can use the binary expansion of $\tilde{v_{k}^{t}}$ and $\widetilde{\tilde{v_{k}}^{t}\tilde{v_{k}}^{t}}$ we obtained for $a_{k}^{t}$, thus we just need one more binary expansion process for the division operation. The required number of bits is

[TABLE]

As for $v_{k}^{t+1}$, we already have binary expansion of $b_{j}^{t}$, then only the division operation needs to be approximated. Note that the projection operator $[\cdot]^{\sqrt{P_{\max}}}_{0}$ needed to generate $v_{k}^{t+1}$ can be implemented by two ReLU units:

[TABLE]

where $I$ is the input and $O$ is projected value of $I$.

The number of bits required for $v_{k}^{t+1}$ is

[TABLE]

Finally, the number of bits we used is at most

[TABLE]

where $n$ satisfies eq. (32).

The number of layers is $1/K^{2}$ times of the number of bits due to parallelism, each bit can be implemented by a constant number of binary units and ReLUs, taking $O$ notation of (36), we get Theorem 2 .