A Bayesian Approach to the Partitioning of Workflows

TL;DR

This paper introduces a Bayesian method using Gibbs Sampling to efficiently estimate processing unit characteristics for optimal workflow partitioning, reducing the need for costly experiments and improving parallel processing efficiency.

Contribution

It presents a novel Bayesian inference approach for workflow partitioning that adapts to uncertain processing unit knowledge, enhancing efficiency over traditional experimental methods.

Findings

Effective estimation of processing unit characteristics using Bayesian inference.

Improved workflow partitioning with lower expected completion time.

Reduced experimental costs in workflow optimization.

Abstract

When partitioning workflows in realistic scenarios, the knowledge of the processing units is often vague or unknown. A naive approach to addressing this issue is to perform many controlled experiments for different workloads, each consisting of multiple number of trials in order to estimate the mean and variance of the specific workload. Since this controlled experimental approach can be quite costly in terms of time and resources, we propose a variant of the Gibbs Sampling algorithm that uses a sequence of Bayesian inference updates to estimate the processing characteristics of the processing units. Using the inferred characteristics of the processing units, we are able to determine the best way to split a workflow for processing it in parallel with the lowest expected completion time and least variance.