When should we trust the annotation? Selective prediction for molecular structure retrieval from mass spectra

TL;DR

This paper presents a selective prediction framework for molecular structure retrieval from mass spectra, allowing models to abstain when uncertainty is high, thereby improving reliability in critical applications.

Contribution

It introduces a risk-coverage based selective prediction method with comprehensive uncertainty quantification strategies evaluated on the MassSpecGym benchmark.

Findings

First-order confidence measures and retrieval-level aleatoric uncertainty perform well.

Fingerprint-level uncertainty scores are poor proxies for success.

Distribution-free risk control enables high-confidence annotation subsets.

Abstract

Machine learning methods for identifying molecular structures from tandem mass spectra (MS/MS) have advanced rapidly, yet current approaches still exhibit significant error rates. In high-stakes applications such as clinical metabolomics and environmental screening, incorrect annotations can have serious consequences, making it essential to determine when a prediction can be trusted. We introduce a selective prediction framework for molecular structure retrieval from MS/MS spectra, enabling models to abstain from predictions when uncertainty is too high. We formulate the problem within the risk-coverage tradeoff framework and comprehensively evaluate uncertainty quantification strategies at two levels of granularity: fingerprint-level uncertainty over predicted molecular fingerprint bits, and retrieval-level uncertainty over candidate rankings. We compare scoring functions including first-order confidence measures, aleatoric and epistemic uncertainty estimates from second-order distributions, as well as distance-based measures in the latent space. All experiments are conducted on the MassSpecGym benchmark. Our analysis reveals that while fingerprint-level uncertainty scores are poor proxies for retrieval success, computationally inexpensive first-order confidence measures and retrieval-level aleatoric uncertainty achieve strong risk-coverage tradeoffs across evaluation settings. We demonstrate that by applying distribution-free risk control via generalization bounds, practitioners can specify a tolerable error rate and obtain a subset of annotations satisfying that constraint with high probability.