sAirflow: Adopting Serverless in a Legacy Workflow Scheduler

TL;DR

sAirflow demonstrates how to adapt a legacy workflow scheduler to serverless environments using minimal code changes, achieving significant cost savings and improved scalability compared to traditional managed solutions.

Contribution

This paper introduces sAirflow, the first serverless adaptation of Airflow's control plane and workers that maintains interface compatibility with minimal code modifications.

Findings

sAirflow halves monetary costs compared to MWAA.

sAirflow scales to 125 workers in seconds, reducing workflow makespan by up to 7x.

sAirflow achieves comparable performance on warm systems.

Abstract

Serverless clouds promise efficient scaling, reduced toil and monetary costs. Yet, serverless-ing a complex, legacy application might require major refactoring and thus is risky. As a case study, we use Airflow, an industry-standard workflow system. To reduce migration risk, we propose to limit code modifications by relying on change data capture (CDC) and message queues for internal communication. To achieve serverless efficiency, we rely on Function-as-a-Service (FaaS). Our system, sAirflow, is the first adaptation of the control plane and workers to the serverless cloud - and it maintains the same interface and most of the code. Experimentally, we show that sAirflow delivers the key serverless benefits: scaling and cost reduction. We compare sAirflow to MWAA, a managed (SaaS) Airflow. On Alibaba benchmarks on warm systems, sAirflow performs similarly while halving the monetary cost. On highly parallel workflows on cold systems, sAirflow scales out in seconds to 125 workers, reducing makespan by 2x-7x.