PingAn: An Insurance Scheme for Job Acceleration in Geo-distributed Big Data Analytics System

TL;DR

PingAn is an online insurance algorithm designed for geo-distributed big data systems that optimizes job completion times by dynamically insuring tasks across clusters, balancing resource use and reliability.

Contribution

It introduces a novel online insurance algorithm with provable performance guarantees for cross-cluster job execution in geo-distributed systems.

Findings

Reduces average job flowtimes by at least 14% compared to state-of-the-art mechanisms.

Achieves up to 40% reduction in job flowtimes in Spark on Yarn.

Demonstrates practicality and effectiveness through trace-driven simulations and real system implementation.

Abstract

Geo-distributed data analysis in a cloud-edge system is emerging as a daily demand. Out of saving time in wide area data transfer, some tasks are dispersed to the edges. However, due to limited computing, overload interference and cluster-level unreachable troubles, efficient execution in the edges is hard, which obstructs the guarantee on the efficiency and reliability of jobs. Launching copies across clusters can be an insurance on a task's completion. Considering cluster heterogeneity and accompanying remote data fetch, cluster selection of copies affects execution quality, as different insuring plans drive different revenues. For providing On-Line-Real-Time analysis results, a system needs to insure the geo-distributed resource for the arriving jobs. Our challenge is to achieve the optimal revenue by dynamically weighing the gains due to insurance against the loss of occupying extra resource for insuring. To this end, we design PingAn, an online insurance algorithm promising $(1+\varepsilon)\!\!-\!speed \ o(\frac{1}{\varepsilon^2+\varepsilon})\!\!-\!competitive$ in sum of the job flowtimes via cross-cluster copying for tasks. PingAn shares resource among the anterior fraction of jobs with the least unprocessed datasize and the fraction is adjustable to fit the system load condition. After sharing, PingAn concretely insures for tasks following efficiency-first reliability-aware principle to optimize the revenue of copies on jobs' performance. Trace-driven simulations demonstrate that PingAn can reduce the average job flowtimes by at least $14\%$ than the state-of-the-art speculation mechanisms. We also build PingAn in Spark on Yarn System to verify its practicality and generality. Experiments show that PingAn can reduce the average job flowtimes by up to $40\%$ comparing to the default Spark execution.