Building Wavelet Histograms on Large Data in MapReduce

TL;DR

This paper presents new algorithms for efficiently constructing wavelet histograms on large datasets using MapReduce, significantly improving performance over existing methods.

Contribution

The paper introduces novel algorithms for exact and approximate wavelet histogram construction optimized for MapReduce environments, demonstrating substantial efficiency gains.

Findings

Significant reduction in computation time and communication costs.

Order-of-magnitude performance improvements over baseline methods.

Effective implementation in Hadoop with large real and synthetic datasets.

Abstract

MapReduce is becoming the de facto framework for storing and processing massive data, due to its excellent scalability, reliability, and elasticity. In many MapReduce applications, obtaining a compact accurate summary of data is essential. Among various data summarization tools, histograms have proven to be particularly important and useful for summarizing data, and the wavelet histogram is one of the most widely used histograms. In this paper, we investigate the problem of building wavelet histograms efficiently on large datasets in MapReduce. We measure the efficiency of the algorithms by both end-to-end running time and communication cost. We demonstrate straightforward adaptations of existing exact and approximate methods for building wavelet histograms to MapReduce clusters are highly inefficient. To that end, we design new algorithms for computing exact and approximate wavelet histograms and discuss their implementation in MapReduce. We illustrate our techniques in Hadoop, and compare to baseline solutions with extensive experiments performed in a heterogeneous Hadoop cluster of 16 nodes, using large real and synthetic datasets, up to hundreds of gigabytes. The results suggest significant (often orders of magnitude) performance improvement achieved by our new algorithms.