# A Two-Phase Dynamic Throughput Optimization Model for Big Data Transfers

## TL;DR

This paper introduces a two-phase dynamic throughput optimization model for big data transfers that combines offline historical analysis with online adaptive decision making, significantly improving transfer efficiency.

## Contribution

It presents a novel two-phase model integrating offline knowledge discovery and online adaptation to optimize data transfer throughput in diverse network conditions.

## Key findings

- Outperformed competitors by up to 1.7x and 5x in different scenarios.
- Achieved up to 93% accuracy compared to optimal throughput.
- Reduced real-time investigation overhead while maintaining high throughput.

## Abstract

The amount of data moved over dedicated and non-dedicated network links increases much faster than the increase in the network capacity, but the current solutions fail to guarantee even the promised achievable transfer throughputs. In this paper, we propose a novel dynamic throughput optimization model based on mathematical modeling with offline knowledge discovery/analysis and adaptive online decision making. In offline analysis, we mine historical transfer logs to perform knowledge discovery about the transfer characteristics. Online phase uses the discovered knowledge from the offline analysis along with real-time investigation of the network condition to optimize the protocol parameters. As real-time investigation is expensive and provides partial knowledge about the current network status, our model uses historical knowledge about the network and data to reduce the real-time investigation overhead while ensuring near optimal throughput for each transfer. Our novel approach is tested over different networks with different datasets and outperformed its closest competitor by 1.7x and the default case by 5x. It also achieved up to 93% accuracy compared with the optimal achievable throughput possible on those networks.