The ATLAS EventIndex: a BigData catalogue for all ATLAS experiment events

TL;DR

The paper presents the design and implementation of the ATLAS EventIndex, a comprehensive BigData catalogue system that manages and facilitates access to all event data collected and generated by the ATLAS experiment at CERN.

Contribution

It introduces a scalable BigData-based architecture for cataloguing and querying vast amounts of particle physics event data, detailing its evolution and future plans.

Findings

Successfully manages billions of event records annually

Enables efficient search and retrieval of specific events

Utilizes open-source BigData tools for system implementation

Abstract

The ATLAS EventIndex system comprises the catalogue of all events collected, processed or generated by the ATLAS experiment at the CERN LHC accelerator, and all associated software tools to collect, store and query this information. ATLAS records several billion particle interactions every year of operation, processes them for analysis and generates even larger simulated data samples; a global catalogue is needed to keep track of the location of each event record and be able to search and retrieve specific events for in-depth investigations. Each EventIndex record includes summary information on the event itself and the pointers to the files containing the full event. Most components of the EventIndex system are implemented using BigData open-source tools. This paper describes the architectural choices and their evolution in time, as well as the past, current and foreseen future implementations of all EventIndex components.