Workflow Partitioning and Deployment on the Cloud using Orchestra

TL;DR

This paper introduces a workflow partitioning method that enhances scalability and performance of cloud-based service workflows by decomposing workflows into sub workflows and deploying them closer to data sources across geographically distributed cloud regions.

Contribution

It proposes a novel workflow partitioning approach that optimizes execution location and parallelism, improving scalability and efficiency over traditional centralized orchestration.

Findings

Significant reduction in network bandwidth usage.

Lower latency in workflow execution.

Improved scalability for large workflows.

Abstract

Orchestrating service-oriented workflows is typically based on a design model that routes both data and control through a single point - the centralised workflow engine. This causes scalability problems that include the unnecessary consumption of the network bandwidth, high latency in transmitting data between the services, and performance bottlenecks. These problems are highly prominent when orchestrating workflows that are composed from services dispersed across distant geographical locations. This paper presents a novel workflow partitioning approach, which attempts to improve the scalability of orchestrating large-scale workflows. It permits the workflow computation to be moved towards the services providing the data in order to garner optimal performance results. This is achieved by decomposing the workflow into smaller sub workflows for parallel execution, and determining the most appropriate network locations to which these sub workflows are transmitted and subsequently executed. This paper demonstrates the efficiency of our approach using a set of experimental workflows that are orchestrated over Amazon EC2 and across several geographic network regions.