Serverless Containers -- rising viable approach to Scientific Workflows

TL;DR

This paper evaluates the use of serverless container platforms like AWS Fargate and Google Cloud Run for scientific workflows, demonstrating their potential benefits and limitations through experimental benchmarks.

Contribution

It extends the HyperFlow engine to support serverless CaaS platforms and provides empirical analysis of their effectiveness for scientific workflows.

Findings

Serverless containers can successfully run scientific workflows.

They offer advantages in elasticity and scalability.

Limitations include potential delays and cost considerations.

Abstract

Increasing popularity of the serverless computing approach has led to the emergence of new cloud infrastructures working in Container-as-a-Service (CaaS) model like AWS Fargate, Google Cloud Run, or Azure Container Instances. They introduce an innovative approach to running cloud containers where developers are freed from managing underlying resources. In this paper, we focus on evaluating capabilities of elastic containers and their usefulness for scientific computing in the scientific workflow paradigm using AWS Fargate and Google Cloud Run infrastructures. For experimental evaluation of our approach, we extended HyperFlow engine to support these CaaS platform, together with adapting four real-world scientific workflows composed of several dozen to over a hundred of tasks organized into a dependency graph. We used these workflows to create cost-performance benchmarks and flow execution plots, measuring delays, elasticity, and scalability. The experiments proved that serverless containers can be successfully applied for scientific workflows. Also, the results allow us to gain insights on specific advantages and limits of such platforms.