A Novel Approach to Translate Structural Aggregation Queries to MapReduce Code

TL;DR

This paper introduces a system that translates array queries with complex structural aggregations from SciDB's AQL into optimized MapReduce code, significantly improving performance and reducing development effort.

Contribution

It presents a novel translator for array queries with structural aggregations, bridging SciDB and MapReduce, and demonstrating performance improvements over handwritten code.

Findings

Generated MapReduce code is up to 10.84x faster than handwritten implementations.

Supports various complex array aggregations like circular, grid, hierarchical, and sliding.

Reduces user effort in developing MapReduce applications for array data.

Abstract

Data management applications are growing and require more attention, especially in the "big data" era. Thus, supporting such applications with novel and efficient algorithms that achieve higher performance is critical. Array database management systems are one way to support these applications by dealing with data represented in n-dimensional data structures. For instance, software like SciDB and RasDaMan can be powerful tools to achieve the required performance on large-scale problems with multidimensional data. Like their relational counterparts, these management systems support specific array query languages as the user interface. As a popular programming model, MapReduce allows large-scale data analysis, facilitates query processing, and is used as a DB engine. Nevertheless, one major obstacle is the low productivity of developing MapReduce applications. Unlike high-level declarative languages such as SQL, MapReduce jobs are written in a low-level descriptive language, often requiring massive programming efforts and complicated debugging processes. This work presents a system that supports translating array queries expressed in the Array Query Language (AQL) in SciDB into MapReduce jobs. We focus on translating some unique structural aggregations, including circular, grid, hierarchical, and sliding aggregations. Unlike traditional aggregations in relational DBs, these structural aggregations are designed explicitly for array manipulation. Thus, our work can be considered an array-view counterpart of existing SQL to MapReduce translators like HiveQL and YSmart. Our translator supports structural aggregations over arrays to meet various array manipulations. The translator can also help user-defined aggregation functions with minimal user effort. We show that our translator can generate optimized MapReduce code, which performs better than the short handwritten code by up to 10.84x.