Scalable Distributed Job Processing with Dynamic Load Balancing

TL;DR

This paper introduces a cost-effective, scalable distributed job processing framework that dynamically balances load, ensures fault tolerance, and adapts to heterogeneous systems without job migration, using a message bus and centralized monitoring.

Contribution

It presents a novel distributed job processing system with integrated load balancing, fault tolerance, and scalability features, avoiding job migration and supporting heterogeneous environments.

Findings

System effectively balances load in heterogeneous environments

Supports fault tolerance and failover recovery

Scales horizontally and vertically to optimize performance

Abstract

We present here a cost effective framework for a robust scalable and distributed job processing system that adapts to the dynamic computing needs easily with efficient load balancing for heterogeneous systems. The design is such that each of the components are self contained and do not depend on each other. Yet, they are still interconnected through an enterprise message bus so as to ensure safe, secure and reliable communication based on transactional features to avoid duplication as well as data loss. The load balancing, fault-tolerance and failover recovery are built into the system through a mechanism of health check facility and a queue based load balancing. The system has a centralized repository with central monitors to keep track of the progress of various job executions as well as status of processors in real-time. The basic requirement of assigning a priority and processing as per priority is built into the framework. The most important aspect of the framework is that it avoids the need for job migration by computing the target processors based on the current load and the various cost factors. The framework will have the capability to scale horizontally as well as vertically to achieve the required performance, thus effectively minimizing the total cost of ownership.