Towards Persistent Memory based Stateful Serverless Computing for Big Data Applications

TL;DR

This paper proposes Marvel, a novel architecture that enables stateful execution and faster I/O for serverless big data applications using persistent memory, significantly improving performance on existing platforms.

Contribution

Marvel introduces a new architecture that supports stateful functions and high-performance storage in serverless environments for big data analytics.

Findings

Reduces execution time of big data applications by up to 86.6%.

Enables stateful function execution with in-memory caching.

Provides faster I/O via PMEM-backed storage.

Abstract

The Function-as-a-service (FaaS) computing model has recently seen significant growth especially for highly scalable, event-driven applications. The easy-to-deploy and cost-efficient fine-grained billing of FaaS is highly attractive to big data applications. However, the stateless nature of serverless platforms poses major challenges when supporting stateful I/O intensive workloads such as a lack of native support for stateful execution, state sharing, and inter-function communication. In this paper, we explore the feasibility of performing stateful big data analytics on serverless platforms and improving I/O throughput of functions by using modern storage technologies such as Intel Optane DC Persistent Memory (PMEM). To this end, we propose Marvel, an end-to-end architecture built on top of the popular serverless platform, Apache OpenWhisk and Apache Hadoop. Marvel makes two main contributions: (1) enable stateful function execution on OpenWhisk by maintaining state information in an in-memory caching layer; and (2) provide access to PMEM backed HDFS storage for faster I/O performance. Our evaluation shows that Marvel reduces the overall execution time of big data applications by up to 86.6% compared to current MapReduce implementations on AWS Lambda.