The xDotGrid Native, Cross-Platform, High-Performance xDFS File Transfer Framework

TL;DR

This paper introduces the high-performance, cross-platform xDFS file transfer protocol, demonstrating its efficiency and comparing it favorably against GridFTP, with implications for distributed systems and cloud applications.

Contribution

The paper presents the design, implementation, and benchmarking of the xDFS protocol, a novel high-performance file transfer system for distributed environments.

Findings

xDFS outperforms GridFTP by 30-53% in throughput for 2GB file transfers

xDFS is cross-platform and supports multi-core architectures

The protocol minimizes overhead and optimizes software design for file transfer

Abstract

In this paper we introduce and describe the highly concurrent xDFS file transfer protocol and examine its cross-platform and cross-language implementation in native code for both Linux and Windows in 32 or 64-bit multi-core processor architectures. The implemented xDFS protocol based on xDotGrid.NET framework is fully compared with the Globus GridFTP protocol. We finally propose the xDFS protocol as a new paradigm of distributed systems for Internet services, and data-intensive Grid and Cloud applications. Also, we incrementally consider different developmental methods of the optimum file transfer systems, and their advantages and disadvantages. The vision of this paper tries as possible to minimize the overhead concerned with the file transfer protocol itself and to examine optimal software design patterns of that protocol. In all disk-to-disk tests for transferring a 2GB file with or without parallelism, the xDFS throughput at minimum 30% and at most 53% was superior to the GridFTP.