Democratizing Scalable Cloud Applications: Transactional Stateful Functions on Streaming Dataflows

TL;DR

This paper introduces a new programming model and streaming dataflow engine that enable scalable, fault-tolerant, and transactional cloud applications, making complex cloud development more accessible and efficient.

Contribution

It presents Stateflow, a high-level programming model, and Styx, a streaming engine supporting serializable transactions and elasticity, advancing cloud application development.

Findings

Styx outperforms existing systems in transaction throughput and latency.

Stateflow simplifies application development with minimal boilerplate.

Styx provides deterministic, fault-tolerant transactions with elastic scaling.

Abstract

Web applications underpin much of modern digital life, yet building scalable and consistent cloud applications remains difficult, requiring expertise across cloud computing, distributed systems, databases, and software engineering. These demands restrict development to a small number of highly specialized engineers. This thesis aims to democratize cloud application development by addressing three challenges: programmability, high-performance fault-tolerant serializable transactions, and serverless semantics. The thesis identifies strong parallels between cloud applications and the streaming dataflow execution model. It first explores this connection through T-Statefun, a transactional extension of Apache Flink Statefun, demonstrating that dataflow systems can support transactional cloud applications via a stateful functions-as-a-service API. However, this approach revealed significant limitations in programmability and performance. To overcome these issues, the thesis introduces Stateflow, a high-level object-oriented programming model that compiles applications into stateful dataflow graphs with minimal boilerplate. Building on this model, the thesis presents Styx, a distributed streaming dataflow engine that provides deterministic, multi-partition, serializable transactions with strong fault tolerance guarantees. Styx eliminates explicit transaction failure handling and significantly outperforms state-of-the-art systems. Finally, the thesis extends Styx with transactional state migration to support elasticity under dynamic workloads.