Data Transfer Optimization Based on Offline Knowledge Discovery and Adaptive Real-time Sampling

TL;DR

This paper introduces a dynamic data transfer optimization model that combines offline knowledge discovery with adaptive real-time sampling to improve throughput efficiency across various networks.

Contribution

The paper presents a novel model integrating offline historical data analysis with online adaptive sampling for network throughput optimization.

Findings

Achieved up to 93% accuracy in throughput prediction.

Reduced real-time network investigation overhead.

Effective across different network types.

Abstract

The amount of data moved over dedicated and non-dedicated network links increases much faster than the increase in the network capacity, but the current solutions fail to guarantee even the promised achievable transfer throughputs. In this paper, we propose a novel dynamic throughput optimization model based on mathematical modeling with offline knowledge discovery/analysis and adaptive online decision making. In offline analysis, we mine historical transfer logs to perform knowledge discovery about the transfer characteristics. Online phase uses the discovered knowledge from the offline analysis along with real-time investigation of the network condition to optimize the protocol parameters. As real-time investigation is expensive and provides partial knowledge about the current network status, our model uses historical knowledge about the network and data to reduce the real-time investigation overhead while ensuring near optimal throughput for each transfer. Our network and data agnostic solution is tested over different networks and achieved up to 93% accuracy compared with the optimal achievable throughput possible on those networks.