Partitioning Uncertain Workflows

TL;DR

This paper introduces a method to partition complex workflows into multiple workloads in distributed environments, reducing overall completion time and variance compared to single-channel execution.

Contribution

The paper presents a novel workflow partitioning method that improves speed and reliability in distributed systems, validated through convex optimization and internet file transmission scenarios.

Findings

Reduced completion time in tested scenarios

Lower variance in workflow completion times

Effective in diverse distributed applications

Abstract

It is common practice to partition complex workflows into separate channels in order to speed up their completion times. When this is done within a distributed environment, unavoidable fluctuations make individual realizations depart from the expected average gains. We present a method for breaking any complex workflow into several workloads in such a way that once their outputs are joined, their full completion takes less time and exhibit smaller variance than when running in only one channel. We demonstrate the effectiveness of this method in two different scenarios; the optimization of a convex function and the transmission of a large computer file over the Internet.