Advancements in Big Data Processing in the ATLAS and CMS Experiments

TL;DR

This paper discusses how big data processing challenges in high-energy physics experiments like ATLAS and CMS are addressed through task splitting, fault tolerance, and reliability engineering to enable petascale data analysis.

Contribution

It introduces new methods for task management and fault recovery in petascale data processing on Grid computing infrastructures, based on experiences from ATLAS and CMS.

Findings

Task splitting enables fine-granularity checkpointing.

Automatic retries improve reliability of large-scale data processing.

Scaling reliability engineering from experiments to broader applications.

Abstract

The ever-increasing volumes of scientific data present new challenges for distributed computing and Grid technologies. The emerging Big Data revolution drives exploration in scientific fields including nanotechnology, astrophysics, high-energy physics, biology and medicine. New initiatives are transforming data-driven scientific fields enabling massive data analysis in new ways. In petascale data processing scientists deal with datasets, not individual files. As a result, a task (comprised of many jobs) became a unit of petascale data processing on the Grid. Splitting of a large data processing task into jobs enabled fine-granularity checkpointing analogous to the splitting of a large file into smaller TCP/IP packets during data transfers. Transferring large data in small packets achieves reliability through automatic re-sending of the dropped TCP/IP packets. Similarly, transient job failures on the Grid can be recovered by automatic re-tries to achieve reliable six sigma production quality in petascale data processing on the Grid. The computing experience of the ATLAS and CMS experiments provides foundation for reliability engineering scaling up Grid technologies for data processing beyond the petascale.