Designing Co-operation in Systems of Hierarchical, Multi-objective Schedulers for Stream Processing
Animesh Dangwal, Yufeng Jiang, Charlie Arnold, Jun Fan, Mohamed Bassem, Aish Rajagopal

TL;DR
This paper presents a novel hierarchical, multi-objective scheduling system for stream processing that enhances load balancing and robustness across complex, large-scale infrastructure, enabling more efficient real-time data processing.
Contribution
It introduces a new design for integrating multiple schedulers in a hierarchical system to improve load balancing and robustness in stream processing infrastructures.
Findings
Enhanced load balancing across compute resources.
Successful integration of multiple schedulers in hierarchy.
Improved robustness for complex applications.
Abstract
Stream processing is a computing paradigm that supports real-time data processing for a wide variety of applications. At Meta, it's used across the company for various tasks such as deriving product insights, providing and improving user services, and enabling AI at scale for our ever-growing user base. Meta's current stream processing framework supports processing TerraBytes(TBs) of data in mere seconds. This is enabled by our efficient schedulers and multi-layered infrastructure, which allocate workloads across various compute resources, working together in hierarchies across various parts of the infrastructure. But with the ever growing complexity of applications, and user needs, areas of the infrastructure that previously required minimal load balancing, now must be made more robust and proactive to application load. In our work we explore how to build and design such a system that…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsReal-Time Systems Scheduling · Advanced Database Systems and Queries · Embedded Systems Design Techniques
