WORKSWORLD: A Domain for Integrated Numeric Planning and Scheduling of Distributed Pipelined Workflows

TL;DR

This paper introduces WORKSWORLD, a domain for automated planning and scheduling of distributed data workflows, enabling efficient construction and execution of complex pipelines across multiple sites.

Contribution

It presents a novel workflow and resource graph representation and a domain for numeric planners to jointly plan and schedule distributed data pipelines.

Findings

Planner solves workflows with up to 14 components across 8 sites

Uses commodity hardware with 1 hour CPU time and 30GB memory

Demonstrates feasibility of automated planning for distributed workflows

Abstract

This work pursues automated planning and scheduling of distributed data pipelines, or workflows. We develop a general workflow and resource graph representation that includes both data processing and sharing components with corresponding network interfaces for scheduling. Leveraging these graphs, we introduce WORKSWORLD, a new domain for numeric domain-independent planners designed for permanently scheduled workflows, like ingest pipelines. Our framework permits users to define data sources, available workflow components, and desired data destinations and formats without explicitly declaring the entire workflow graph as a goal. The planner solves a joint planning and scheduling problem, producing a plan that both builds the workflow graph and schedules its components on the resource graph. We empirically show that a state-of-the-art numeric planner running on commodity hardware with one hour of CPU time and 30GB of memory can solve linear-chain workflows of up to 14 components across eight sites.