Data Diffusion: Dynamic Resource Provision and Data-Aware Scheduling for Data Intensive Applications

TL;DR

This paper introduces a data diffusion approach that dynamically manages resources, data replication, and scheduling to enhance efficiency and performance in data-intensive applications, especially under variable workloads.

Contribution

It proposes a novel data diffusion model integrating resource provisioning, data replication, and locality-aware scheduling validated on real-world applications.

Findings

Performance index increased by up to 34X

Application response time improved by over 506X

Achieved near-optimal throughput and execution times

Abstract

Data intensive applications often involve the analysis of large datasets that require large amounts of compute and storage resources. While dedicated compute and/or storage farms offer good task/data throughput, they suffer low resource utilization problem under varying workloads conditions. If we instead move such data to distributed computing resources, then we incur expensive data transfer cost. In this paper, we propose a data diffusion approach that combines dynamic resource provisioning, on-demand data replication and caching, and data locality-aware scheduling to achieve improved resource efficiency under varying workloads. We define an abstract "data diffusion model" that takes into consideration the workload characteristics, data accessing cost, application throughput and resource utilization; we validate the model using a real-world large-scale astronomy application. Our results show that data diffusion can increase the performance index by as much as 34X, and improve application response time by over 506X, while achieving near-optimal throughputs and execution times.