Combining Serverless and High-Performance Computing Paradigms to support ML Data-Intensive Applications

TL;DR

This paper presents Cylon, a high-performance distributed data frame system that combines serverless and HPC paradigms, enabling efficient large-scale data processing in cloud environments with near-HPC performance.

Contribution

The paper introduces a novel serverless communicator for distributed data processing, achieving HPC-like efficiency on AWS Lambda for data-intensive applications.

Findings

Scaling efficiency within 6.5% of serverful AWS at 64 nodes

Demonstrates effective direct communication via NAT Traversal TCP Hole Punching

Shows promising results for Python-based data processing in serverless environments

Abstract

Data is found everywhere, from health and human infrastructure to the surge of sensors and the proliferation of internet-connected devices. To meet this challenge, the data engineering field has expanded significantly in recent years in both research and industry. Traditionally, data engineering, Machine Learning, and AI workloads have been run on large clusters within data center environments, requiring substantial investment in hardware and maintenance. With the rise of the public cloud, it is now possible to run large applications across nodes without owning or maintaining hardware. Serverless functions such as AWS Lambda provide horizontal scaling and precise billing without the hassle of managing traditional cloud infrastructure. However, when processing large datasets, users often rely on external storage options that are significantly slower than direct communication typical of HPC clusters. We introduce Cylon, a high-performance distributed data frame solution that has shown promising results for data processing using Python. We describe how we took inspiration from the FMI library and designed a serverless communicator to tackle communication and performance issues associated with serverless functions. With our design, we demonstrate that the scaling efficiency of AWS Lambda achieves within 6.5% of serverful AWS (EC2) at 64 nodes, based on implementing direct communication via NAT Traversal TCP Hole Punching.