Task Allocation for Distributed Stream Processing

TL;DR

This paper formalizes the task allocation problem in distributed stream processing, proves its NP-hardness, and proposes an approximation algorithm for a broad class of applications, addressing a key theoretical gap.

Contribution

It introduces a formal NP-hardness proof for task allocation and presents a novel approximation algorithm for series-parallel graphs in stream processing.

Findings

The task allocation problem is NP-hard.

The proposed algorithm achieves a constant-factor approximation.

The algorithm performs well when resources scale logarithmically with tasks.

Abstract

There is a growing demand for live, on-the-fly processing of increasingly large amounts of data. In order to ensure the timely and reliable processing of streaming data, a variety of distributed stream processing architectures and platforms have been developed, which handle the fundamental tasks of (dynamically) assigning processing tasks to the currently available physical resources and routing streaming data between these resources. However, while there are plenty of platforms offering such functionality, the theory behind it is not well understood. In particular, it is unclear how to best allocate the processing tasks to the given resources. In this paper, we establish a theoretical foundation by formally defining a task allocation problem for distributed stream processing, which we prove to be NP-hard. Furthermore, we propose an approximation algorithm for the class of series-parallel decomposable graphs, which captures a broad range of common stream processing applications. The algorithm achieves a constant-factor approximation under the assumptions that the number of resources scales at least logarithmically with the number of computational tasks and the computational cost of the tasks dominates the cost of communication.