Scalable Querying of Nested Data

TL;DR

This paper introduces a scalable framework for efficiently querying nested data collections in distributed systems, addressing performance and load balancing issues through program translation, optimized data representation, and skew handling.

Contribution

It presents a novel framework that converts nested collection programs into shredded queries, improving performance and load balancing in distributed query processing.

Findings

Significant performance improvements in diverse nested collection scenarios

Effective automated skew handling for load balancing

Enhanced query evaluation efficiency with the proposed framework

Abstract

While large-scale distributed data processing platforms have become an attractive target for query processing, these systems are problematic for applications that deal with nested collections. Programmers are forced either to perform non-trivial translations of collection programs or to employ automated flattening procedures, both of which lead to performance problems. These challenges only worsen for nested collections with skewed cardinalities, where both handcrafted rewriting and automated flattening are unable to enforce load balancing across partitions. In this work, we propose a framework that translates a program manipulating nested collections into a set of semantically equivalent shredded queries that can be efficiently evaluated. The framework employs a combination of query compilation techniques, an efficient data representation for nested collections, and automated skew-handling. We provide an extensive experimental evaluation, demonstrating significant improvements provided by the framework in diverse scenarios for nested collection programs.