AI Enabled Data Quality Monitoring with Hydra

TL;DR

This paper presents Hydra, an AI-based data quality monitoring system for physics experiments, demonstrating rapid development, deployment challenges, and initial successful results in a production environment.

Contribution

It introduces Hydra, a novel AI-enabled monitoring system that leverages off-the-shelf technologies for efficient data quality assessment in physics experiments.

Findings

Hydra was successfully deployed in production at Gluex.

Early results show improved fault detection capabilities.

Sociological challenges in AI system deployment are discussed.

Abstract

Data quality monitoring is critical to all experiments impacting the quality of any physics results. Traditionally, this is done through an alarm system, which detects low level faults, leaving higher level monitoring to human crews. Artificial Intelligence is beginning to find its way into scientific applications, but comes with difficulties, relying on the acquisition of new skill sets, either through education or acquisition, in data science. This paper will discuss the development and deployment of the Hydra monitoring system in production at Gluex. It will show how "off-the-shelf" technologies can be rapidly developed, as well as discuss what sociological hurdles must be overcome to successfully deploy such a system. Early results from production running of Hydra will also be shared as well as a future outlook for development of Hydra.