Dirigo: Self-scaling Stateful Actors For Serverless Real-time Data Processing

TL;DR

Dirigo introduces a serverless distributed stream processing system using virtual actors, achieving resource efficiency and performance isolation through dynamic autoscaling and fine-grained scheduling for real-time data processing.

Contribution

It presents a novel actor abstraction and data plane scheduling mechanism that enable self-scaling and resource sharing in serverless stream processing.

Findings

High resource efficiency demonstrated through time-sharing of compute resources.

Improved performance isolation via function autoscaling.

Effective message-level scheduling enabling fine-grained control.

Abstract

We propose Dirigo, a distributed stream processing service built atop virtual actors. Dirigo achieves both a high level of resource efficiency and performance isolation driven by user intent (SLO). To improve resource efficiency, Dirigo adopts a serverless architecture that enables time-sharing of compute resources among streaming operators, both within and across applications. Meanwhile, Dirigo improves performance isolation by inheriting the property of function autoscaling from serverless architecture. Specifically, Dirigo proposes (i) dual-mode actor, an actor abstraction that dynamically provides orderliness guarantee for streaming operator during autoscaling and (ii) a data plane scheduling mechanism, along with its API, that allows scheduling and scaling at the message-level granularity.