Federating distributed storage for clouds in ATLAS

TL;DR

This paper discusses the implementation and integration of a federated storage system, Dynafed, at CERN for the ATLAS experiment, enhancing data location and retrieval across distributed cloud repositories.

Contribution

It presents the deployment of Dynafed for ATLAS, supporting multiple protocols and simplifying data access through federation in a large-scale scientific environment.

Findings

Successful setup and integration of Dynafed with ATLAS data system

First operational experiences with federated data access in ATLAS

Identification of challenges in deployment and usage

Abstract

Input data for applications that run in cloud computing centres can be stored at distant repositories, often with multiple copies of the popular data stored at many sites. Locating and retrieving the remote data can be challenging, and we believe that federating the storage can address this problem. A federation would locate the closest copy of the data on the basis of GeoIP information. Currently we are using the dynamic data federation Dynafed, a software solution developed by CERN IT. Dynafed supports several industry standards for connection protocols like Amazon's S3, Microsoft's Azure, as well as WebDAV and HTTP. Dynafed functions as an abstraction layer under which protocol-dependent authentication details are hidden from the user, requiring the user to only provide an X509 certificate. We have setup an instance of Dynafed and integrated it into the ATLAS data distribution management system. We report on the challenges faced during the installation and integration. We have tested ATLAS analysis jobs submitted by the PanDA production system and we report on our first experiences with its operation.