The GA4GH Task Execution API: Enabling Easy Multi Cloud Task Execution

TL;DR

The paper introduces the GA4GH Task Execution API, a standardized interface enabling seamless task execution across diverse compute environments, facilitating federated data analysis and multi-cloud workflows in genomics research.

Contribution

It presents a flexible, extensible API standard for multi-cloud and hybrid compute task management, adopted by multiple providers and workflow engines.

Findings

Supports diverse compute environments including HPC, cloud, and hybrid systems.

Enables federated and distributed data analysis workflows.

Facilitates load balancing across multi-cloud infrastructures.

Abstract

The Global Alliance for Genomics and Health (GA4GH) Task Execution Service (TES) API is a standardized schema and API for describing and executing batch execution tasks. It provides a common way to submit and manage tasks to a variety of compute environments, including on premise High Performance Compute and High Throughput Computing (HPC/HTC) systems, Cloud computing platforms, and hybrid environments. The TES API is designed to be flexible and extensible, allowing it to be adapted to a wide range of use cases, such as "bringing compute to the data" solutions for federated and distributed data analysis or load balancing across multi cloud infrastructures. This API has been adopted by a number of different service providers and utilized by several workflow engines. Using its capabilities, genomes research institutes are building hybrid compute systems to study life science.