Signal Confidence Limits from a Neural Network Data Analysis

TL;DR

This paper develops a method to determine confidence limits for signal detection in neural network data analysis, even when signal and background counts are unknown, providing a statistical framework similar to Clopper-Pearson bounds.

Contribution

It introduces a way to calculate confidence bounds for signal likelihood from neural network tagged data, extending classical statistical bounds to complex neural network scenarios.

Findings

Confidence bounds for signal likelihood are derived from neural network tagged data.

The method parallels Clopper-Pearson bounds used in direct signal detection.

Challenges are identified in applying Bayesian methods with maximum entropy priors.

Abstract

This paper deals with a situation of some importance for the analysis of experimental data via Neural Network (NN) or similar devices: Let $N$ data be given, such that $N=N_s+N_b$, where $N_s$ is the number of signals, $N_b$ the number of background events, both unknown. Assume that a NN has been trained, such that it will tag signals with efficiency $F_s$, $(0<F_s<1)$ and background data with $F_b$, $(0<F_b<1)$. Applying the NN yields $N^Y$ tagged events. We demonstrate that the knowledge of $N^Y$ is sufficient to calculate confidence bounds for the signal likelihood, which have the same statistical interpretation as the Clopper-Pearson bounds for the well-studied case of direct signal observation. Subsequently, we discuss rigorous bounds for the a-posteriori distribution function of the signal probability, as well as for the (closely related) likelihood that there are $N_s$ signals in the data. We compare them with results obtained by starting off with a maximum entropy type assumption for the a-priori likelihood that there are $N_s$ signals in the data and applying the Bayesian theorem. Difficulties are encountered with the latter method.