Investigations of the Systematic Uncertainties in Convolutional Neural Network Based Analysis of Atmospheric Cherenkov Telescope Data

TL;DR

This study evaluates the stability of convolutional neural networks in atmospheric Cherenkov telescope data analysis, highlighting their robustness under various conditions but noting sensitivity to camera noise levels.

Contribution

It systematically investigates how observational and instrumental variations affect CNN performance in Cherenkov telescope data analysis, emphasizing the importance of including noise considerations in training.

Findings

Most systematics are comparable to traditional methods.

CNN predictions are highly sensitive to camera noise, with up to 50% variation in gamma-ray acceptance.

Performance stability depends on accounting for observational effects during training.

Abstract

Machine learning, through the use of convolutional and recurrent neural networks is a promising avenue for the improvement of background rejection performance in imaging atmospheric Cherenkov telescopes. However, it is of paramount importance for science analysis that their performance remains stable against a wide range of observing conditions and instrument states. We investigate the stability of convolutional recurrent networks by applying them to background rejection in a toy Monte Carlo simulation of a Cherenkov telescope array. We then vary a range of observation and instrument parameters in the simulation. In general, most of the resulting systematics are at a level not much greater than conventional analyses. However, a strong dependence of the neural network predictions on the noise level within the camera was found, with differences of up to 50% in the gamma-ray acceptance rate in very noisy environments. It is clear from the performance differences seen in these studies that these observational effects must be considered in the training step of the final analysis when using such networks for background rejection in Cherenkov telescope observations.