Dirigo: Self-scaling Stateful Actors For Serverless Real-time Data Processing
Le Xu, Divyanshu Saxena, Neeraja J. Yadwadkar, Aditya Akella, and, Indranil Gupta

TL;DR
Dirigo introduces a serverless distributed stream processing system using virtual actors, achieving resource efficiency and performance isolation through dynamic autoscaling and fine-grained scheduling for real-time data processing.
Contribution
It presents a novel actor abstraction and data plane scheduling mechanism that enable self-scaling and resource sharing in serverless stream processing.
Findings
High resource efficiency demonstrated through time-sharing of compute resources.
Improved performance isolation via function autoscaling.
Effective message-level scheduling enabling fine-grained control.
Abstract
We propose Dirigo, a distributed stream processing service built atop virtual actors. Dirigo achieves both a high level of resource efficiency and performance isolation driven by user intent (SLO). To improve resource efficiency, Dirigo adopts a serverless architecture that enables time-sharing of compute resources among streaming operators, both within and across applications. Meanwhile, Dirigo improves performance isolation by inheriting the property of function autoscaling from serverless architecture. Specifically, Dirigo proposes (i) dual-mode actor, an actor abstraction that dynamically provides orderliness guarantee for streaming operator during autoscaling and (ii) a data plane scheduling mechanism, along with its API, that allows scheduling and scaling at the message-level granularity.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsCloud Computing and Resource Management · Distributed and Parallel Computing Systems · IoT and Edge/Fog Computing
