Uncertainty Quantification for Named Entity Recognition via Full-Sequence and Subsequence Conformal Prediction

TL;DR

This paper introduces a conformal prediction framework for NER models that provides uncertainty-aware prediction sets with formal coverage guarantees, improving reliability in NLP applications.

Contribution

It develops a general, efficient method to produce uncertainty estimates for NER models with finite-sample coverage guarantees, adaptable to various conditions.

Findings

Prediction sets achieve valid coverage across datasets.

Method supports both unconditional and class-conditional coverage.

Framework accounts for heterogeneity in sentence and entity characteristics.

Abstract

Named Entity Recognition (NER) serves as a foundational component in many natural language processing (NLP) pipelines. However, current NER models typically output a single predicted label sequence without any accompanying measure of uncertainty, leaving downstream applications vulnerable to cascading errors. In this paper, we introduce a general framework for adapting sequence-labeling-based NER models to produce uncertainty-aware prediction sets. These prediction sets are collections of full-sentence labelings that are guaranteed to contain the correct labeling with a user-specified confidence level. This approach serves a role analogous to confidence intervals in classical statistics by providing formal guarantees about the reliability of model predictions. Our method builds on conformal prediction, which offers finite-sample coverage guarantees under minimal assumptions. We design efficient nonconformity scoring functions to construct efficient, well-calibrated prediction sets that support both unconditional and class-conditional coverage. This framework accounts for heterogeneity across sentence length, language, entity type, and number of entities within a sentence. Empirical experiments on four NER models across three benchmark datasets demonstrate the broad applicability, validity, and efficiency of the proposed methods.