DPA Load Balancer: Load balancing for Data Parallel Actor-based systems

TL;DR

This paper presents DPA Load Balancer, a dynamic load balancing method for actor-based streaming systems that mitigates data skew and stragglers through continuous monitoring, input redistribution, and state merging.

Contribution

It introduces a novel load balancing approach combining input forwarding and state merging to address data skew in actor-based streaming frameworks.

Findings

Reduces stragglers caused by data skew

Effective load redistribution via consistent hashing

Improved processing efficiency in streaming systems

Abstract

In this project we explore ways to dynamically load balance actors in a streaming framework. This is used to address input data skew that might lead to stragglers. We continuously monitor actors' input queue lengths for load, and redistribute inputs among reducers using consistent hashing if we detect stragglers. To ensure consistent processing post-redistribution, we adopt an approach that uses input forwarding combined with a state merge step at the end of the processing. We show that this approach can greatly alleviate stragglers for skewed data.