MorphStream: Scalable Processing of Transactions over Streams on Multicores

TL;DR

MorphStream is a novel transactional stream processing engine that enhances parallelism and dynamic scheduling on multicores, significantly improving performance and supporting complex transactional features.

Contribution

It introduces a three-stage execution paradigm enabling dynamic scheduling and parallel processing in TSPEs, a largely unexplored area.

Findings

Outperforms existing TSPEs in various scenarios

Supports windowed state transactions

Handles non-deterministic state access

Abstract

Transactional Stream Processing Engines (TSPEs) form the backbone of modern stream applications handling shared mutable states. Yet, the full potential of these systems, specifically in exploiting parallelism and implementing dynamic scheduling strategies, is largely unexplored. We present MorphStream, a TSPE designed to optimize parallelism and performance for transactional stream processing on multicores. Through a unique three-stage execution paradigm (i.e., planning, scheduling, and execution), MorphStream enables dynamic scheduling and parallel processing in TSPEs. Our experiment showcased MorphStream outperforms current TSPEs across various scenarios and offers support for windowed state transactions and non-deterministic state access, demonstrating its potential for broad applicability.