Making Heads and Tails of Models with Marginal Calibration for Sparse Tagsets

TL;DR

This paper investigates calibration of probabilistic tagging models with sparse tagsets, proposing methods to measure and improve calibration accuracy across different tag frequency groups, enhancing model reliability.

Contribution

It introduces tag frequency grouping (TFG) for measuring calibration error and demonstrates effective recalibration strategies for sequence taggers with sparse tagsets.

Findings

Post-hoc recalibration reduces calibration error.

TFG effectively measures calibration across frequency bands.

Separate group recalibration improves calibration equity.

Abstract

For interpreting the behavior of a probabilistic model, it is useful to measure a model's calibration--the extent to which it produces reliable confidence scores. We address the open problem of calibration for tagging models with sparse tagsets, and recommend strategies to measure and reduce calibration error (CE) in such models. We show that several post-hoc recalibration techniques all reduce calibration error across the marginal distribution for two existing sequence taggers. Moreover, we propose tag frequency grouping (TFG) as a way to measure calibration error in different frequency bands. Further, recalibrating each group separately promotes a more equitable reduction of calibration error across the tag frequency spectrum.