Benchmarking XRootD-HTTPS on 400Gbps Links with Variable Latencies

TL;DR

This paper evaluates the performance of XRootD-HTTPS data transfers over 400 Gbps links with variable latencies, aiming to inform infrastructure improvements for high-bandwidth scientific data access.

Contribution

It provides a systematic benchmarking of XRootD-HTTPS performance under realistic high-bandwidth, high-latency network conditions, highlighting necessary hardware and software optimizations.

Findings

XRootD-HTTPS can sustain 400 Gbps transfers with proper configurations

Performance varies significantly with latency and host setup

Optimal CPU and origin configurations improve transfer efficiency

Abstract

In anticipation of the High Luminosity-LHC era, there is a critical need to oversee software readiness for upcoming growth in network traffic for production and user data analysis access. This paper looks into software and hardware required improvements in US-CMS Tier-2 sites to be able to sustain and meet the projected 400 Gbps bandwidth demands while tackling the challenge posed by varying latencies between sites. Specifically, our study focuses on identifying the performance of XRootD HTTP third-party copies across multiple 400 Gbps links and exploring different host and transfer configurations. Our approach involves systematic testing with variations in the number of origins per cluster and CPU allocations for each origin. By replicating real network conditions and creating network "loops" that traverse multiple switches across the wide area network, we are able to replicate authentic network conditions