Accelerating Large-scale Data Exploration through Data Diffusion

TL;DR

This paper introduces a data diffusion approach that dynamically acquires resources, replicates data, and schedules computations near data to improve large-scale data exploration efficiency and scalability.

Contribution

It presents a novel data diffusion method that adapts resource allocation and data placement in response to demand, enhancing performance without high hardware costs.

Findings

Improves performance over alternative approaches.

Scales linearly with the number of data cache nodes.

Effective in large-scale astronomy data analysis.

Abstract

Data-intensive applications often require exploratory analysis of large datasets. If analysis is performed on distributed resources, data locality can be crucial to high throughput and performance. We propose a "data diffusion" approach that acquires compute and storage resources dynamically, replicates data in response to demand, and schedules computations close to data. As demand increases, more resources are acquired, thus allowing faster response to subsequent requests that refer to the same data; when demand drops, resources are released. This approach can provide the benefits of dedicated hardware without the associated high costs, depending on workload and resource characteristics. The approach is reminiscent of cooperative caching, web-caching, and peer-to-peer storage systems, but addresses different application demands. Other data-aware scheduling approaches assume dedicated resources, which can be expensive and/or inefficient if load varies significantly. To explore the feasibility of the data diffusion approach, we have extended the Falkon resource provisioning and task scheduling system to support data caching and data-aware scheduling. Performance results from both micro-benchmarks and a large scale astronomy application demonstrate that our approach improves performance relative to alternative approaches, as well as provides improved scalability as aggregated I/O bandwidth scales linearly with the number of data cache nodes.