PlinyCompute: A Platform for High-Performance, Distributed, Data-Intensive Tool Development

TL;DR

PlinyCompute is a high-performance, distributed data-intensive system offering a high-level declarative interface and a low-level object model, enabling efficient development of reusable tools with significant speed advantages over Spark.

Contribution

It introduces a hybrid system combining declarative optimization with a custom memory model, outperforming JVM-based systems like Spark in data-intensive computations.

Findings

Achieves 2x to over 50x speedup compared to Spark.

Provides a high-level declarative interface with a custom object model.

Demonstrates efficient development of reusable data tools.

Abstract

This paper describes PlinyCompute, a system for development of high-performance, data-intensive, distributed computing tools and libraries. In the large, PlinyCompute presents the programmer with a very high-level, declarative interface, relying on automatic, relational-database style optimization to figure out how to stage distributed computations. However, in the small, PlinyCompute presents the capable systems programmer with a persistent object data model and API (the "PC object model") and associated memory management system that has been designed from the ground-up for high performance, distributed, data-intensive computing. This contrasts with most other Big Data systems, which are constructed on top of the Java Virtual Machine (JVM), and hence must at least partially cede performance-critical concerns such as memory management (including layout and de/allocation) and virtual method/function dispatch to the JVM. This hybrid approach---declarative in the large, trusting the programmer's ability to utilize PC object model efficiently in the small---results in a system that is ideal for the development of reusable, data-intensive tools and libraries. Through extensive benchmarking, we show that implementing complex objects manipulation and non-trivial, library-style computations on top of PlinyCompute can result in a speedup of 2x to more than 50x or more compared to equivalent implementations on Spark.