Application Level High Speed Transfer Optimization Based on Historical Analysis and Real-time Tuning

TL;DR

This paper introduces HARP, a predictive algorithm that combines historical data analysis and real-time probing to optimize data transfer protocols, significantly improving throughput in dynamic network environments.

Contribution

The paper presents a novel hybrid approach for data transfer tuning that outperforms existing methods by leveraging both historical and real-time network data.

Findings

HARP achieves up to 50% higher throughput than existing solutions.

Combining historical analysis with real-time probing reduces overhead and improves accuracy.

Experimental results validate the effectiveness of HARP across various network conditions.

Abstract

Data-intensive scientific and commercial applications increasingly require frequent movement of large datasets from one site to the other(s). Despite growing network capacities, these data movements rarely achieve the promised data transfer rates of the underlying physical network due to poorly tuned data transfer protocols. Accurately and efficiently tuning the data transfer protocol parameters in a dynamically changing network environment is a major challenge and remains as an open research problem. In this paper, we present predictive end-to-end data transfer optimization algorithms based on historical data analysis and real-time background traffic probing, dubbed HARP. Most of the previous work in this area are solely based on real time network probing which results either in an excessive sampling overhead or fails to accurately predict the optimal transfer parameters. Combining historical data analysis with real time sampling enables our algorithms to tune the application level data transfer parameters accurately and efficiently to achieve close-to-optimal end-to-end data transfer throughput with very low overhead. Our experimental analysis over a variety of network settings shows that HARP outperforms existing solutions by up to 50% in terms of the achieved throughput.