Monte Carlo conformal prediction for quantifying uncertainty in radio galaxy classification under ambiguous ground truth

TL;DR

This paper evaluates Monte Carlo conformal prediction as a method to quantify uncertainty in radio galaxy classification, comparing it with Bayesian deep learning uncertainty measures, and finds weak correlation between them.

Contribution

It introduces MCCP for astronomical classification uncertainty quantification and compares its effectiveness with Bayesian entropy measures.

Findings

MCCP provides a different uncertainty measure than Bayesian methods.

Weak correlation observed between MCCP set sizes and Bayesian entropy.

Calibrated MCCP can be applied to radio galaxy classification tasks.

Abstract

Dramatically increasing data volumes are forcing astronomers to adopt automated methods for the identification and classification of astronomical objects. Although deep-learning models are often well-suited to this task, obtaining a measure of uncertainty on their predictions is challenging. Here we consider the suitability of Monte Carlo conformal prediction (MCCP) set size and confidence as measures of model uncertainty for the astronomical classification of radio galaxies. We demonstrate this approach using model predictions from a pre-trained radio galaxy foundation model, fine-tuned on a smaller set of labelled radio galaxies. We calibrate the MCCP by obtaining annotator-derived soft label distributions, i.e. probability distributions over classes instead of single class assignments, for each of these labelled radio galaxies and compare the resulting set sizes and confidence scores to predictive entropy measures for each galaxy obtained using a supervised Bayesian deep-learning model trained using Hamiltonian Monte Carlo (HMC). The comparison reveals only a weak correlation between the measures.