Occupy the Cloud: Distributed Computing for the 99%

TL;DR

This paper advocates for using stateless functions as a simple, elastic platform for distributed computing, making it more accessible and efficient for a broad range of users and applications.

Contribution

It introduces PyWren, a prototype demonstrating how stateless functions can replace complex cluster management and support various distributed computing models.

Findings

PyWren efficiently implements BSP model

Stateless functions reduce cluster management overhead

Future data processing can leverage disaggregated storage

Abstract

Distributed computing remains inaccessible to a large number of users, in spite of many open source platforms and extensive commercial offerings. While distributed computation frameworks have moved beyond a simple map-reduce model, many users are still left to struggle with complex cluster management and configuration tools, even for running simple embarrassingly parallel jobs. We argue that stateless functions represent a viable platform for these users, eliminating cluster management overhead, fulfilling the promise of elasticity. Furthermore, using our prototype implementation, PyWren, we show that this model is general enough to implement a number of distributed computing models, such as BSP, efficiently. Extrapolating from recent trends in network bandwidth and the advent of disaggregated storage, we suggest that stateless functions are a natural fit for data processing in future computing environments.