Classification uncertainty for transient gravitational-wave noise artefacts with optimised conformal prediction

TL;DR

This paper applies conformal prediction to quantify uncertainty in glitch classification for gravitational wave data, optimizing nonconformity measures to improve reliability of ML-based classifications.

Contribution

It demonstrates the use of conformal prediction in gravitational wave glitch classification and optimizes nonconformity measures for better uncertainty quantification.

Findings

Optimal nonconformity measure varies with application and metric

Conformal prediction provides rigorous uncertainty estimates for ML classifiers

Application improves reliability of glitch classification in gravitational wave data

Abstract

With the increasing use of Machine Learning (ML) algorithms in scientific research comes the need for reliable uncertainty quantification. When taking a measurement it is not enough to provide the result, we also have to declare how confident we are in the measurement. This is also true when the results are obtained from a ML algorithm, and arguably more so since the internal workings of ML algorithms are often less transparent compared to traditional statistical methods. Additionally, many ML algorithms do not provide uncertainty estimates and auxiliary algorithms must be applied. Conformal Prediction (CP) is a framework to provide such uncertainty quantifications for ML point predictors. In this paper, we explore the use and properties of CP applied in the context of glitch classification in gravitational wave astronomy. Specifically, we demonstrate the application of CP to the Gravity Spy glitch classification algorithm. CP makes use of a score function, a nonconformity measure, to convert an algorithm's heuristic notion of uncertainty to a rigorous uncertainty. We use the application on Gravity Spy to explore the performance of different nonconformity measures and optimise them for our application. Our results show that the optimal nonconformity measure depends on the specific application, as well as the metric used to quantify the performance.