Streaming Big Data meets Backpressure in Distributed Network Computation

TL;DR

This paper analyzes the performance limits of distributed network computation under streaming big data workloads, proposing adaptive algorithms based on backpressure to optimize resource allocation and maximize query throughput.

Contribution

It extends backpressure algorithms to incorporate computation over query streams, providing a universal, distributed approach for resource allocation in networked data processing.

Findings

Algorithms support maximum sustainable query rate.

Performance limits characterized by communication and computation constraints.

Framework applicable to network clouds and fog computing environments.

Abstract

We study network response to queries that require computation of remotely located data and seek to characterize the performance limits in terms of maximum sustainable query rate that can be satisfied. The available resources include (i) a communication network graph with links over which data is routed, (ii) computation nodes, over which computation load is balanced, and (iii) network nodes that need to schedule raw and processed data transmissions. Our aim is to design a universal methodology and distributed algorithm to adaptively allocate resources in order to support maximum query rate. The proposed algorithms extend in a nontrivial way the backpressure (BP) algorithm to take into account computations operated over query streams. They contribute to the fundamental understanding of network computation performance limits when the query rate is limited by both the communication bandwidth and the computation capacity, a classical setting that arises in streaming big data applications in network clouds and fogs.