Data Movement Manager (DMM) for the SENSE-Rucio Interoperation Prototype

TL;DR

The paper presents the Data Movement Manager (DMM), a prototype that integrates CERN's Rucio with SDN services to optimize high-energy physics data transfers through priority-based bandwidth allocation and detailed flow monitoring.

Contribution

It introduces a novel interface connecting Rucio with SDN, enabling optimized, monitored data flows in the LHC computing grid infrastructure.

Findings

Enables SDN-enabled data flows for high-energy physics

Implements transfer priority-based bandwidth allocation

Provides detailed flow performance monitoring

Abstract

The Data Movement Manager (DMM) is a prototype interface that connects CERN's data management software, Rucio, with the Sofware-Defined Networking (SDN) service SENSE by ESNet. It enables SDN-enabled high-energy physics data flows using the existing worldwide LHC computing grid infrastructure. A key feature of DMM is transfer priority-based bandwidth allocation, optimizing network usage. Additionally, it provides fine-grained monitoring of underperforming flows by leveraging end-to-end data flow monitoring. This is achieved through access to host-level (network interface) throughput metrics and transfer-tool (FTS) data transfer job-level metrics. This paper details the design and implementation of DMM.