Hybrid Reusable Computational Analytics Workflow Management with Cloudmesh

TL;DR

This paper presents a minimalistic, hybrid workflow management framework called Cloudmesh that simplifies coordinating computational analytics across diverse resources, including local, HPC, and cloud systems, with multiple user interfaces.

Contribution

The paper introduces a flexible, easy-to-use workflow system that integrates local, HPC, and cloud resources, supporting educational and benchmarking applications.

Findings

Framework supports multiple resource types and interfaces

Demonstrated on AI benchmarks and educational use cases

Open-source code available for community use

Abstract

In this paper, we summarize our effort to create and utilize a simple framework to coordinate computational analytics tasks with the help of a workflow system. Our design is based on a minimalistic approach while at the same time allowing to access computational resources offered through the owner's computer, HPC computing centers, cloud resources, and distributed systems in general. The access to this framework includes a simple GUI for monitoring and managing the workflow, a REST service, a command line interface, as well as a Python interface. The resulting framework was developed for several examples targeting benchmarks of AI applications on hybrid compute resources and as an educational tool for teaching scientists and students sophisticated concepts to execute computations on resources ranging from a single computer to many thousands of computers as part of on-premise and cloud infrastructure. We demonstrate the usefulness of the tool on a number of examples. The code is available as an open-source project in GitHub and is based on an easy-to-enhance tool called cloudmesh.