DPA Load Balancer: Load balancing for Data Parallel Actor-based systems
Ziheng Wang, Atem Aguer, and Amir Ziai

TL;DR
This paper presents DPA Load Balancer, a dynamic load balancing method for actor-based streaming systems that mitigates data skew and stragglers through continuous monitoring, input redistribution, and state merging.
Contribution
It introduces a novel load balancing approach combining input forwarding and state merging to address data skew in actor-based streaming frameworks.
Findings
Reduces stragglers caused by data skew
Effective load redistribution via consistent hashing
Improved processing efficiency in streaming systems
Abstract
In this project we explore ways to dynamically load balance actors in a streaming framework. This is used to address input data skew that might lead to stragglers. We continuously monitor actors' input queue lengths for load, and redistribute inputs among reducers using consistent hashing if we detect stragglers. To ensure consistent processing post-redistribution, we adopt an approach that uses input forwarding combined with a state merge step at the end of the processing. We show that this approach can greatly alleviate stragglers for skewed data.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Data Storage Technologies · Cloud Computing and Resource Management · Data Stream Mining Techniques
