Push vs. Pull-Based Loop Fusion in Query Engines

TL;DR

This paper compares push-based and pull-based query engines, introduces a new hybrid engine inspired by loop fusion, and evaluates their performance, showing the hybrid approach combines advantages and performs best in microbenchmarks.

Contribution

It presents a novel pull-based engine inspired by loop fusion techniques, bridging the gap between existing approaches and providing a fair evaluation environment.

Findings

No significant advantage for either engine in realistic workloads.

The proposed hybrid engine outperforms existing engines in microbenchmarks.

Fair evaluation shows traditional biases affected previous comparisons.

Abstract

Database query engines use pull-based or push-based approaches to avoid the materialization of data across query operators. In this paper, we study these two types of query engines in depth and present the limitations and advantages of each engine. Similarly, the programming languages community has developed loop fusion techniques to remove intermediate collections in the context of collection programming. We draw parallels between the DB and PL communities by demonstrating the connection between pipelined query engines and loop fusion techniques. Based on this connection, we propose a new type of pull-based engine, inspired by a loop fusion technique, which combines the benefits of both approaches. Then we experimentally evaluate the various engines, in the context of query compilation, for the first time in a fair environment, eliminating the biasing impact of ancillary optimizations that have traditionally only been used with one of the approaches. We show that for realistic analytical workloads, there is no considerable advantage for either form of pipelined query engine, as opposed to what recent research suggests. Also, by using microbenchmarks we show that our proposed engine dominates the existing engines by combining the benefits of both.