To Ship or Not to (Function) Ship (Extended version)

TL;DR

This paper compares function shipping and data shipping for interactive big data query processing with sampling, analyzing when each method is preferable based on data transfer, CPU load, sampling method, and query volume.

Contribution

It provides a detailed analysis of the conditions under which data shipping outperforms function shipping in interactive query scenarios with sampling.

Findings

Data shipping can be up to 6.5x faster with clustered sampling and high CPU utilization.

The choice between function and data shipping depends on data transfer volume, CPU load, sampling technique, and query frequency.

The study offers guidelines for optimizing query processing strategies in big data analytics.

Abstract

Sampling is often used to reduce query latency for interactive big data analytics. The established parallel data processing paradigm relies on function shipping, where a coordinator dispatches queries to worker nodes and then collects the results. The commoditization of high-performance networking makes data shipping possible, where the coordinator directly reads data in the workers' memory using RDMA while workers process other queries. In this work, we explore when to use function shipping or data shipping for interactive query processing with sampling. Whether function shipping or data shipping should be preferred depends on the amount of data transferred, the current CPU utilization, the sampling method and the number of queries executed over the data set. The results show that data shipping is up to 6.5x faster when performing clustered sampling with heavily-utilized workers.