AgileDART: An Agile and Scalable Edge Stream Processing Engine

TL;DR

AgileDart is a novel edge stream processing engine that uses dynamic dataflow and bandit-based path planning to achieve low latency, scalability, and adaptability in heterogeneous and dynamic edge environments.

Contribution

It introduces a dynamic dataflow abstraction and a bandit-based path planning model tailored for scalable, low-latency edge stream processing.

Findings

AgileDart reduces query latency compared to Storm and EdgeWise.

It significantly improves scalability for many concurrent edge queries.

It adapts effectively to workload variations and network unreliability.

Abstract

Edge applications generate a large influx of sensor data on massive scales, and these massive data streams must be processed shortly to derive actionable intelligence. However, traditional data processing systems are not well-suited for these edge applications as they often do not scale well with a large number of concurrent stream queries, do not support low-latency processing under limited edge computing resources, and do not adapt to the level of heterogeneity and dynamicity commonly present in edge computing environments. As such, we present AgileDart, an agile and scalable edge stream processing engine that enables fast stream processing of many concurrently running low-latency edge applications' queries at scale in dynamic, heterogeneous edge environments. The novelty of our work lies in a dynamic dataflow abstraction that leverages distributed hash table-based peer-to-peer overlay networks to autonomously place, chain, and scale stream operators to reduce query latencies, adapt to workload variations, and recover from failures and a bandit-based path planning model that re-plans the data shuffling paths to adapt to unreliable and heterogeneous edge networks. We show that AgileDart outperforms Storm and EdgeWise on query latency and significantly improves scalability and adaptability when processing many real-world edge stream applications' queries.