DotDFS: A Grid-based high-throughput file transfer system

TL;DR

DotDFS is a high-throughput, open-standard file transfer protocol designed for Grid infrastructures, outperforming GridFTP in LAN tests and offering a new parallel TCP paradigm for efficient distributed file transfers.

Contribution

The paper introduces DotDFS, a novel high-throughput file transfer protocol with a new parallel TCP approach, enhancing performance and cross-platform compatibility in Grid environments.

Findings

Better LAN transfer rates than GridFTP in multiple TCP streams.

Accesses 94% of bottleneck bandwidth in memory-to-memory tests.

Identifies technical issues in GridFTP's protocol and implementation.

Abstract

DotGrid platform is a Grid infrastructure integrated with a set of open and standard protocols recently implemented on the top of Microsoft .NET in Windows and MONO .NET in UNIX/Linux. DotGrid infrastructure along with its proposed protocols provides a right and solid approach to targeting other platforms, e.g., the native C/C++ runtime. In this paper, we propose a new file transfer protocol called DotDFS as a high-throughput distributed file transfer component for DotGrid. DotDFS introduces some open binary protocols for efficient file transfers on current Grid infrastructures. DotDFS protocol also provides mechanisms for multiple file streams to gain high-throughput file transfer similar to GridFTP protocol, but by proposing and implementing a new parallel TCP connection-oriented paradigm. In our LAN tests, we have achieved better results than Globus GridFTP implementation particularly in multiple TCP streams and directory tree transfers. Our LAN experiences in memory-to-memory tests show that DotDFS accesses to the 94% bottleneck bandwidth while GridFTP is accessing 91%. In LAN disk-to-disk tests, comparing DotDFS protocol with GridFTP protocol unveils a set of interesting and technical problems in GridFTP for both the nature of the protocol and its implementation by Globus. In the WAN experimental studies, we propose a new idea for analytical modeling of file transfer protocols like DotDFS inspired by sampling, experimentation and mathematical interpolation approaches. The cross-platform and open standard-based features of DotDFS provide a substantial framework for unifying data access and resource sharing in real heterogeneous Grid environments.