Unified Approach for Weakly Supervised Multicalibration

TL;DR

This paper introduces a unified framework and algorithms for multicalibration in weakly supervised learning, enabling reliable uncertainty estimates without needing clean labels.

Contribution

It develops estimators and correction methods for multicalibration in weak supervision, along with a post-hoc recalibration algorithm called WLMC.

Findings

Effective multicalibration correction in weak supervision settings

Finite-sample guarantees for the proposed estimators

Empirical insights into uncertainty estimation without clean labels

Abstract

Multicalibration requires predicted scores to agree with label probabilities across rich families of subgroups and score-dependent tests, but existing methods require clean input-label pairs for evaluation and post-processing. This assumption fails in weakly supervised learning (WSL) regimes -- including positive-unlabeled, unlabeled-unlabeled, and positive-confidence learning -- where clean labels are costly or unavailable even though reliable uncertainty estimates may be crucial. We address this gap by developing estimators of multicalibration error and post-hoc correction methods for WSL settings in which clean input-label pairs are unavailable. We propose a unified framework for estimating and correcting multicalibration under weak supervision by combining contamination-matrix risk rewrites with witness-based calibration constraints, yielding corrected multicalibration moments with finite-sample guarantees. We further propose weak-label multicalibration boost (WLMC), a generic post-hoc recalibration algorithm under weak supervision. Finally, we conduct experiments across multiple weak-supervision settings to evaluate multicalibration behavior and offer empirical insight into uncertainty estimation under weak supervision.