Stream Processing With Dependency-Guided Synchronization (Extended Version)

TL;DR

This paper introduces dependency-guided synchronization (DGS), a new programming model for stream processing that improves parallelism and performance in complex synchronization scenarios, outperforming existing APIs in popular systems.

Contribution

The paper proposes DGS, a novel synchronization model for stateful streaming computations, enabling better parallelism and scalability compared to traditional APIs.

Findings

DGS enables scalable performance in stream processing applications.

Synchronization plans improve throughput over existing APIs.

Programming overhead with DGS is minimal.

Abstract

Real-time data processing applications with low latency requirements have led to the increasing popularity of stream processing systems. While such systems offer convenient APIs that can be used to achieve data parallelism automatically, they offer limited support for computations that require synchronization between parallel nodes. In this paper, we propose *dependency-guided synchronization (DGS)*, an alternative programming model for stateful streaming computations with complex synchronization requirements. In the proposed model, the input is viewed as partially ordered, and the program consists of a set of parallelization constructs which are applied to decompose the partial order and process events independently. Our programming model maps to an execution model called *synchronization plans* which supports synchronization between parallel nodes. Our evaluation shows that APIs offered by two widely used systems -- Flink and Timely Dataflow -- cannot suitably expose parallelism in some representative applications. In contrast, DGS enables implementations with scalable performance, the resulting synchronization plans offer throughput improvements when implemented manually in existing systems, and the programming overhead is small compared to writing sequential code.