Distributed Programming via Safe Closure Passing

TL;DR

This paper introduces a new asynchronous distributed programming model that emphasizes safety, simplicity, and performance by passing closures to immutable data, supported by static type-based optimizations, with a prototype in Scala.

Contribution

It proposes a novel functional programming model for distributed systems that ensures safe closure passing through type restrictions and demonstrates its implementation and preliminary performance results.

Findings

Type-based restrictions improve serialization efficiency.

Prototype implementation in Scala validates the model.

Preliminary results show promising performance benefits.

Abstract

Programming systems incorporating aspects of functional programming, e.g., higher-order functions, are becoming increasingly popular for large-scale distributed programming. New frameworks such as Apache Spark leverage functional techniques to provide high-level, declarative APIs for in-memory data analytics, often outperforming traditional "big data" frameworks like Hadoop MapReduce. However, widely-used programming models remain rather ad-hoc; aspects such as implementation trade-offs, static typing, and semantics are not yet well-understood. We present a new asynchronous programming model that has at its core several principles facilitating functional processing of distributed data. The emphasis of our model is on simplicity, performance, and expressiveness. The primary means of communication is by passing functions (closures) to distributed, immutable data. To ensure safe and efficient distribution of closures, our model leverages both syntactic and type-based restrictions. We report on a prototype implementation in Scala. Finally, we present preliminary experimental results evaluating the performance impact of a static, type-based optimization of serialization.