Creating a content delivery network for general science on the internet backbone using XCaches

TL;DR

This paper presents a novel content delivery network using XRootD caches on the internet backbone to improve data delivery efficiency for scientific workflows across various domains.

Contribution

It introduces an integrated cache-based CDN for scientific data, enhancing CPU efficiency and reducing bandwidth for diverse scientific workflows.

Findings

Increased CPU efficiency for scientific workflows.

Reduced network bandwidth consumption.

Effective data reuse across multiple science domains.

Abstract

A general problem faced by computing on the grid for opportunistic users is that delivering cycles is simpler than delivering data to those cycles. In this project we show how we integrated XRootD caches placed on the internet backbone to implement a content delivery network for general science workflows. We will show that for some workflows on different science domains like high energy physics, gravitational waves, and others the combination of data reuse from the workflows together with the use of caches increases CPU efficiency while decreasing network bandwidth use.