Data intensive physics analysis in Azure cloud

TL;DR

This paper discusses how UCSD leveraged Azure cloud resources, integrated with the Open Science Grid, to enhance data-intensive physics analysis for the CMS experiment, reducing analysis time and improving resource flexibility.

Contribution

It demonstrates the integration of commercial cloud resources with existing scientific grid infrastructure for data-intensive physics analysis.

Findings

Successful deployment of Azure cloud resources for CMS data analysis

Enhanced data caching infrastructure improved job efficiency

Reduced time to results for physics analyses

Abstract

The Compact Muon Solenoid (CMS) experiment at the Large Hadron Collider (LHC) is one of the largest data producers in the scientific world, with standard data products centrally produced, and then used by often competing teams within the collaboration. This work is focused on how a local institution, University of California San Diego (UCSD), partnered with the Open Science Grid (OSG) to use Azure cloud resources to augment its available computing to accelerate time to results for multiple analyses pursued by a small group of collaborators. The OSG is a federated infrastructure allowing many independent resource providers to serve many independent user communities in a transparent manner. Historically the resources would come from various research institutions, spanning small universities to large HPC centers, based on either community needs or grant allocations, so adding commercial clouds as resource providers is a natural evolution. The OSG technology allows for easy integration of cloud resources, but the data-intensive nature of CMS compute jobs required the deployment of additional data caching infrastructure to ensure high efficiency.