Towards Big data processing in IoT: network management for online edge data processing
Shuo Wan, Jiaxun Lu, Pingyi Fan, and Khaled B. Letaief

TL;DR
This paper proposes a Lyapunov optimization-based network management algorithm for MEC in IoT, balancing data processing delay and energy consumption for efficient big data handling at the edge.
Contribution
It introduces a novel online resource management algorithm that jointly optimizes edge processing and transmission parameters without prior data distribution knowledge.
Findings
The algorithm stabilizes data processing delay.
It reduces energy consumption in edge networks.
It effectively manages limited edge computing resources.
Abstract
Heavy data load and wide cover range have always been crucial problems for internet of things (IoT). However, in mobile-edge computing (MEC) network, the huge data can be partly processed at the edge. In this paper, a MEC-based big data analysis network is discussed. The raw data generated by distributed network terminals are collected and processed by edge servers. The edge servers split out a large sum of redundant data and transmit extracted information to the center cloud for further analysis. However, for consideration of limited edge computation ability, part of the raw data in huge data sources may be directly transmitted to the cloud. To manage limited resources online, we propose an algorithm based on Lyapunov optimization to jointly optimize the policy of edge processor frequency, transmission power and bandwidth allocation. The algorithm aims at stabilizing data processing…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsIoT and Edge/Fog Computing · IoT Networks and Protocols · Energy Efficient Wireless Sensor Networks
Towards Big data processing in IoT: network management for online edge data processing
Shuo Wan, Jiaxun Lu, Pingyi Fan, and Khaled B. Letaief*,
Tsinghua National Laboratory for Information Science and Technology(TNList),
Department of Electronic Engineering, Tsinghua University, Beijing, P.R. China
E-mail: [email protected], [email protected], [email protected]
*Department of Electronic Engineering, Hong Kong University of Science and Technology, Hong Kong
Email: [email protected]
Abstract
Heavy data load and wide cover range have always been crucial problems for internet of things (IoT). However, in mobile-edge computing (MEC) network, the huge data can be partly processed at the edge. In this paper, a MEC-based big data analysis network is discussed. The raw data generated by distributed network terminals are collected and processed by edge servers. The edge servers split out a large sum of redundant data and transmit extracted information to the center cloud for further analysis. However, for consideration of limited edge computation ability, part of the raw data in huge data sources may be directly transmitted to the cloud. To manage limited resources online, we propose an algorithm based on Lyapunov optimization to jointly optimize the policy of edge processor frequency, transmission power and bandwidth allocation. The algorithm aims at stabilizing data processing delay and saving energy without knowing probability distributions of data sources. The proposed network management algorithm may contribute to big data processing in future IoT.
Index Terms:
Internet of things, Big data, Edge computing, Network management
I Introduction
The internet of things (IoT) has emerged as a huge network, which extends connected agents beyond standard devices to any range of traditionally non-internet-enabled devices. In IoT, a large range of everyday objects such as vehicles, home appliances and street lamps may all enter the network and exchange data. The extension will result in an extraordinary increase of data amount and network cover range, which is far beyond the capability of the existing network. Recently, Mobile-edge computing (MEC) has emerged as a promising technique in IoT. By deploying cloud-like infrastructure in the vicinity of edge devices, a large proportion of computing load can be distributed to the edge [1].
In the literature, the problem of computation offloading, network resource allocation and related network structure designs in MEC have been broadly studied in various models [2, 3, 4, 5, 6, 7]. In [2], the authors employed deep reinforcement learning to allocate caching, computing and communication resources for MEC system in vehicle networks. In [3], the authors optimized the offload decision and resource allocation to obtain a maximum computation rate for a wireless powered MEC system. Considering the combination of MEC and existing communication service, a novel two-layer TDMA-based unified resource management scheme was proposed to handle both conventional communication service and MEC data traffic at the same time [4]. In [5], the authors jointly optimized the radio and computational resource for Multi-user MEC computing system. In [6], notions of energy harvesting were further considered. In addition to the edge, the cloud was also taken into consideration in [7].
The MEC system design considering computation task offloading has been sufficiently investigated in previous works. However, for IoT big data processing, MEC server may also serves to process local data at the edge [8, 9, 10]. In [8], the authors discussed the application of MEC in data processing. In [9], the authors indicated that edge servers can process part of the data rather than completely deliver them to the cloud. Then in [10], the authors proposed a scheme for this system. In the field of edge computing, the algorithm design for distributed data processing is still an open problem.
In this paper, we consider an MEC-based distributed data processing system as shown in Fig .1. In this system, servers at the network edge collect data from around data sources and conduct initial steps of data processing. Consider the common redundancy in raw data [11], the edge processing will wipe out a large amount of redundant data and transmit extracted information to the cloud. It is assumed that the extracted information takes only a little bandwidth to transmit. As the edge processing speed is limited, part of the raw data will be transmitted to the cloud in cases of high data rate. As the communication resources are also limited, the rest data will be temporarily stored, which results in waiting delay.
Based on Lyapunov optimization, we proposed an algorithm to derive an online policy of network management. Without knowing probability distributions of arriving data, it can smartly manage network sources to stabilize delay while saving energy. When data rate reduces, edge servers can lower down their processor frequency to save energy. In cases of high data rate, data offloading assists to raise edge processing speed. Furthermore, the allocation of bandwidth for data offloading can also adjust the edge processing capability based on their buffer lengths. In condition of high data rate, the smart design of bandwidth allocation can further stabilize edge processing delay. In order to figure out the policy design, we propose a network management algorithm based on Lyapunov optimization.
II System model
We consider an IoT network for online data collection and analysis. The data sources are distributed in a wide range. The data are supposed to be generated randomly and transmitted to IoT edge servers. The distributed edge processing results are sent to center cloud for further analysis. The IoT network management policy is determined per time slot. The edge servers are represented by , where index belongs to set and the discrete time slot set is denoted as . In this section, we will introduce the model of data collection and processing.
II-A Data collection
The widely distributed network devices generate data indicating local information. Edge servers collect data from their around devices. It is supposed that edge server collects bits data during time slot , where and . The collected data will be temporarily stored in a data buffer for processing. Suppose the edge server is able to deal with bits data in time slot . Its data buffer length is updated by
[TABLE]
It is assumed that are independent among different devices and different time slots. is supposed to satisfy poisson distribution with . Besides,for consideration of rate limitation in practical network, it is supposed that is bounded by . That is, any larger than will be cut as .
II-B Edge computation model
It is assumed edge server has the capability to deal with bits data in time slot . Among the bits data, bits data are processed locally by edge server and bits data are transmitted to center. It is assumed that the edge servers will split out a large sum of redundant data and the extracted results take only a small proportion of bandwidth for transmission. Furthermore, the limited edge processing speed may not catch up with upcoming data rate. Then a large proportion of bandwidth can be allocated to for offloading data.
II-B1 Edge data processing
It is assumed that the edge server needs CPU cycles to precess one bit data, which depends on the applied algorithm [5]. The processor cycle frequency of at time is denoted as with . Then is
[TABLE]
where is the time slot length. The power consumption rate of edge data processing [12] by is
[TABLE]
where is the effective switched capacitance [12] of , which is determined by chip structure.
II-B2 Data transmission model
The edge data processing is limited by edge processor and energy resources. To lower down the delay, the network communication bandwidth is allocated to edge servers for transmission of collected data. It is assumed that the wireless channels between edge servers and center cloud are i.i.d. frequency-flat block fading [13]. Thus the channel power gain between edge server and center cloud can be denoted by , where is the small-scale fading channel power gain, is the pass loss constant, is the pass loss exponent, is reference distance and is the distance between and center cloud. Under the application of FDMA, by Shannon formula [14], the data transmission capacity between and center cloud in time slot is
[TABLE]
where is the proportion of the bandwidth allocated to , is the transmission power with , is the entire bandwidth for data transmission and is the noise power spectral density. is the bandwidth allocation vector at time with and .
III Problem formulation
The data offloading policy focus on the power consumption with respect to edge data processing and data transmission. In time slot , the power consumption of edge processing of is denoted as . The data transmission power of in time slot is . Then the power consumption of in time slot is
[TABLE]
Then the average weighted sum power consumption is
[TABLE]
where is a positive parameter with regard to edge server , which can be adjusted to balance power management of all edge nodes. As the system performance metrics, is the long-term edge power consumption. The data offloading policy with respect to can be derived by statistical optimization.
The data collected by edge servers will be temporarily stored in a data buffer. In this case, the data queuing delay is the metrics of edge system service quality. By Little’s Law [15], the average queuing delay of a queuing agent is proportional to the average queuing length. Therefore, the average data amount in edge data memory is viewed as the system service quality metrics. The long-term queuing length for edge server is defined as
[TABLE]
The network management policy in time slot is denoted as . The operation set is the processor frequency of edge servers. The operation set is the transmission power of data offloading. The parameter is the set of bandwidth allocation policy. Therefore, the optimal policy design is formulated as follows.
[TABLE]
Eq .(8a) is the bandwidth allocation constraint, where is a system constant. Constraints (8b) indicates the bound of edge processor frequency and data transmission power. Index belongs to set and time slot belongs to set . For delay consideration, constraint (8c) forces the edge data buffers to be stable, which guarantees the collected data can be processed in a finite delay.
The proposed is obviously a statistical optimization problem with randomly arriving data. Therefore, the policy has to be determined dynamically in each time slot. Furthermore, the spatial coupling of bandwidth allocation among edge servers induces great challenge to the problem solution. Instead of solving directly, we propose an online jointly resource management algorithm based on Lyapunov optimization.
IV Online network management
IV-A Lyapunov optimization framework
The proposed is a challenging statistical optimization problem. By Lyapunov optimization [16], is formulated as a deterministic problem for each time slot, which can be solved with low complexity. The online algorithm can cope with the dynamical random environment while deriving an overall optimal outcome. Based on Lyapunov optimization framework ,the algorithm aims at saving energy while stabilizing the edge data buffers.
For online resource management, the Lyapunov function for each time slot is defined as
[TABLE]
This quadratic function is a scalar measure of data accumulation in queue. Then the Lyapunov drift is defined as follows.
[TABLE]
To stabilize the network queuing buffer while minimizing the average energy penalty, the data processing policy is determined by minimizing a bound on the following drift-plus-penalty function for each time slot .
[TABLE]
where is a positive system parameter which represents the tradeoff between Lyapunov drift and energy cost. is the expectation of a random process whose probability distribution is supposed to be unknown. Therefore, an upper bound of is estimated so that we can minimize without the specific probability distribution. According to the following Lemma 1, we derive a deterministic upper bound of for each time slot.
Lemma 1**.**
For an arbitrary policy constrained by (8a), (8b) and (8c), the Lyapunov drift function is upper bounded by
[TABLE]
where is a known constant independent with the system policy and is the current data buffer length. is the edge processing data bits amount while is the offloaded data amount. They are all for time slot .
Proof.
From equation (1), we have
[TABLE]
By (13), we can subtract on both side and sum up the inequalities for , which leads to follows.
[TABLE]
It has been stated that is bounded by . Note that the computation and communication resources are limited, then and are also bounded by their corresponding maximum rate. As the maximum processor frequency is , we have . Since and , we have . For simplicity, we separately denote and as and . Then the term should be bounded by Therefore, we have
[TABLE]
where . When considering a specific time slot , it is straightforward to see that is a deterministic constant. This completes the proof. ∎
Together with (11) and (12), the drift-plus penalty function is upper-bounded by
[TABLE]
By optimizing the above upper bound of in each time slot , the data queuing length can be stabilized on a low level while the power consumption penalty is also minimized. In Lemma 1, parameter is not affected by system policy. Therefore, it is reasonable to omit in the policy determination problem.
Then the modified problem is defined as follows.
[TABLE]
IV-B Solution for
In last subsection, we formulated for deriving optimal policy in each time slot. The optimization objectives include edge computation processor frequency , data transmission power and bandwidth allocation . In this section, we will divide into two subproblems and derive a solution for optimal policy.
IV-B1 Optimal frequency for edge processor
We first delete part of the objective function which is not related with . Then it is straightforward to see that the subproblem with respect to is defined as follows.
[TABLE]
It is obvious to confirm that is a convex optimization problem. Furthermore, there is no coupling among elements in . Therefore, the optimal processor frequency can be derived separately for each edge server. The stationary point of is . In addition, the optimal processor frequency may also be the boundary . Then the final solution is given by
[TABLE]
IV-B2 Bandwidth allocation and data transmission power
We reserve the elements with respect to and and derive the following subproblem.
[TABLE]
In (20), we have
[TABLE]
Note that this is a perspective function of with . It is straightforward to see that is a concave function with respect to . Then is a jointly concave function with respect to and . Therefore, is a convex optimization problem. Though it can be solved directly by conventional solvers, the dimensional curse may still be a large obstacle. In this paper, we employ an iterative algorithm to solve in a more efficient way.
Suppose the bandwidth allocation is fixed, a sub-problem can be derived as follows.
[TABLE]
As is fixed, in are independent with each other. Therefore, we can separately obtain with respect to each index . The stationary point of system cost function is . Considering constraint (22a), the optimal solution of is
[TABLE]
Suppose is figured out, a sub-problem to optimize is derived as follows.
[TABLE]
In this case, elements in are coupled with each other. Therefore, we employ dual decomposition [17]. For , the Lagrange function is
[TABLE]
To decouple , we set
[TABLE]
Then we have , the dual sub-problem of is
[TABLE]
Dual sub-problem can be solved by gradient decent method. The corresponding gradient is , where achieves the lower bound in (IV-B2). As stated before, this is a convex optimization problem. Therefore, either the stationary point or achieves the lower bound. The value of stationary point is the zero point of the derivative function, which can be derived by bisection method. By iteratively updating and corresponding , we can finally derive the optimal bandwidth allocation and transmission power.
In summary, the optimal policy for online edge data processing can be illustrated by chart in Fig .2.
V Simulation results
We consider simulations of a network composed of edge servers and a center cloud. It is assumed that the center cloud has an equal distance with edge servers, which is set as 200. satisfies poisson distribution with rate . Besides, for consideration of maximum data collection speed in real system, is constrained in . In simulations, if the randomly generated is larger than , it will be set as . The small scale fading channel power gain is generated by exponential distribution . Besides, other parameter sets include , , , , , , , , , , , and .
The system performance in terms of power consumption and edge buffer length is first tested by two network management strategies. The results are shown in Fig .3. The curves marked by squares are obtained by evenly allocating bandwidth. Except for , its and are both optimized. That is, the bandwidth allocation is separated from computation resource management. Curves without marks are obtained by optimizing bandwidth allocation. Fig .3 shows that the average power consumption monotonically decreases with respect to control parameter . Fig .3 shows that the average edge buffer length monotonically increases with respect to . By (19) and (23), the increase of will decrease and , which reduces energy consumption while lowering down the edge processing speed. Meanwhile, as shown in Fig .3 and Fig .3, increasing data rate results in the rise of power consumption and edge buffer length. However, the performance deteriorates when taking evenly bandwidth allocation. This shows the importance of jointly optimizing bandwidth allocation and computation resources.
Fig .4 shows the average edge buffer length with respect to time. The curve marked by square is obtained by evenly bandwidth allocation. In cases of high data rate, the strategy with optimal bandwidth allocation achieves a stable edge buffer length. By Little’s Law, this indicates a stable data processing delay, which is crucial for online data processing. However, the strategy with evenly bandwidth allocation shows an awful performance. In cases of high data rate with randomly arriving data amount, optimal bandwidth allocation tend to allocate more resources to edges with larger data amount. This explains its performance of stabilizing edge buffer length. Therefore, it is important to jointly consider bandwidth allocation in network management.
VI Conclusion
In this paper, we investigated network management strategies for online edge data processing in IoT. Focused on saving energy while stabilizing delay, an online MEC-based network management algorithm was proposed based on Lyapunov optimization framework. In cases of low data rate, edge processor frequency and transmission power are dynamically reduced for saving energy. In cases of high data rate, the bandwidth resources are optimally allocated for stabilizing data processing delay. By theoretical analysis and simulation tests, we validated the performance of the proposed dynamical network management algorithm with respect to system design parameters. The online policy is obtained by current data buffer length regardless of data source probability distributions.
The reference list from the paper itself. Each links out to its DOI / PubMed record.
- 1[1] B. P. Rimal, D. P. Van, and M. Maier, “Cloudlet enhanced fiber-wireless access networks for mobile-edge computing,” IEEE Transactions on Wireless Communications , vol. 16, no. 6, pp. 3601–3618, June 2017.
- 2[2] Y. He, N. Zhao, and H. Yin, “Integrated networking, caching, and computing for connected vehicles: A deep reinforcement learning approach,” IEEE Transactions on Vehicular Technology , vol. 67, no. 1, pp. 44–55, 2018.
- 3[3] S. Bi and Y. J. Zhang, “Computation rate maximization for wireless powered mobile-edge computing with binary computation offloading,” IEEE Transactions on Wireless Communications , vol. 17, no. 6, pp. 4177–4190, 2018.
- 4[4] B. P. Rimal, D. P. Van, and M. Maier, “Cloudlet enhanced fiber-wireless access networks for mobile-edge computing,” IEEE Transactions on Wireless Communications , vol. 16, no. 6, pp. 3601–3618, 2017.
- 5[5] Y. Mao, J. Zhang, S. Song, and K. B. Letaief, “Stochastic joint radio and computational resource management for multi-user mobile-edge computing systems,” IEEE Transactions on Wireless Communications , vol. 16, no. 9, pp. 5994–6009, 2017.
- 6[6] H. Zheng, K. Xiong, P. Fan, Z. Zhong, and K. B. Letaief, “Fog-assisted multi-user swipt networks: Local computing or offloading,” IEEE Internet of Things Journal , 2019.
- 7[7] S.-H. Park, O. Simeone, and S. Shamai, “Joint optimization of cloud and edge processing for fog radio access networks,” in Information Theory (ISIT), 2016 IEEE International Symposium on . IEEE, 2016, pp. 315–319.
- 8[8] W. Shi, J. Cao, Q. Zhang, Y. Li, and L. Xu, “Edge computing: Vision and challenges,” IEEE Internet of Things Journal , vol. 3, no. 5, pp. 637–646, 2016.
