Globus Automation Services: Research process automation across the space-time continuum

TL;DR

This paper introduces new Globus automation services that enable reliable, scalable, and secure automation of complex research workflows across diverse, distributed scientific resources and environments.

Contribution

The paper presents a novel cloud-based platform with flexible, event-driven automation capabilities for research processes involving heterogeneous and long-lived resources.

Findings

Supports long-lived, reliable execution despite failures

Provides a flexible API for diverse actions and flows

Enables secure, event-driven automation across distributed resources

Abstract

Research process automation -- the reliable, efficient, and reproducible execution of linked sets of actions on scientific instruments, computers, data stores, and other resources -- has emerged as an essential element of modern science. We report here on new services within the Globus research data management platform that enable the specification of diverse research processes as reusable sets of actions, \emph{flows}, and the execution of such flows in heterogeneous research environments. To support flows with broad spatial extent (e.g., from scientific instrument to remote data center) and temporal extent (from seconds to weeks), these Globus automation services feature: 1) cloud hosting for reliable execution of even long-lived flows despite sporadic failures; 2) a simple specification and extensible asynchronous action provider API, for defining and executing a wide variety of actions and flows involving heterogeneous resources; 3) an event-driven execution model for automating execution of flows in response to arbitrary events; and 4) a rich security model enabling authorization delegation mechanisms for secure execution of long-running actions across distributed resources. These services permit researchers to outsource and automate the management of a broad range of research tasks to a reliable, scalable, and secure cloud platform. We present use cases for Globus automation services, describe their design and implementation, present microbenchmark studies, and review experiences applying the services in a range of applications.