$\lambda$FS: A Scalable and Elastic Distributed File System Metadata Service using Serverless Functions

TL;DR

$ lambda$FS is a scalable, elastic distributed file system metadata service built on serverless functions, significantly improving throughput, latency, and cost efficiency for large-scale DFS operations.

Contribution

The paper introduces $ lambda$FS, a novel serverless-based metadata service architecture that overcomes scalability and performance limitations of traditional MDS systems.

Findings

Achieves up to 4.13x higher throughput

Reduces latency by 90.4%

Lowers cost by 86%

Abstract

The metadata service (MDS) sits on the critical path for distributed file system (DFS) operations, and therefore it is key to the overall performance of a large-scale DFS. Common "serverful" MDS architectures, such as a single server or cluster of servers, have a significant shortcoming: either they are not scalable, or they make it difficult to achieve an optimal balance of performance, resource utilization, and cost. A modern MDS requires a novel architecture that addresses this shortcoming. To this end, we design and implement $\lambda$FS, an elastic, high-performance metadata service for large-scale DFSes. $\lambda$FS scales a DFS metadata cache elastically on a FaaS (Function-as-a-Service) platform and synthesizes a series of techniques to overcome the obstacles that are encountered when building large, stateful, and performance-sensitive applications on FaaS platforms. $\lambda$FS takes full advantage of the unique benefits offered by FaaS $\unicode{x2013}$ elastic scaling and massive parallelism $\unicode{x2013}$ to realize a highly-optimized metadata service capable of sustaining up to 4.13$\times$ higher throughput, 90.40% lower latency, 85.99% lower cost, 3.33$\times$ better performance-per-cost, and better resource utilization and efficiency than a state-of-the-art DFS for an industrial workload.