A Confidence-Variance Theory for Pseudo-Label Selection in Semi-Supervised Learning

TL;DR

This paper introduces a Confidence-Variance (CoVar) framework for pseudo-label selection in semi-supervised learning, combining confidence and residual-class variance to improve reliability over traditional fixed-threshold methods.

Contribution

It develops a novel reliability measure based on confidence and variance, and integrates it into semi-supervised methods, outperforming fixed-threshold strategies across multiple datasets.

Findings

Consistent improvement over strong baselines on multiple datasets.

A threshold-free pseudo-label selection mechanism based on confidence-variance.

Enhanced reliability in pseudo-label selection leading to better semi-supervised learning performance.

Abstract

Most pseudo-label selection strategies in semi-supervised learning rely on fixed confidence thresholds, implicitly assuming that prediction confidence reliably indicates correctness. In practice, deep networks are often overconfident: high-confidence predictions can still be wrong, while informative low-confidence samples near decision boundaries are discarded. This paper introduces a Confidence-Variance (CoVar) theory framework that provides a principled joint reliability criterion for pseudo-label selection. Starting from the entropy minimization principle, we derive a reliability measure that combines maximum confidence (MC) with residual-class variance (RCV), which characterizes how probability mass is distributed over non-maximum classes. The derivation shows that reliable pseudo-labels should have both high MC and low RCV, and that the influence of RCV increases as confidence grows, thereby correcting overconfident but unstable predictions. From this perspective, we cast pseudo-label selection as a spectral relaxation problem that maximizes separability in a confidence-variance feature space, and design a threshold-free selection mechanism to distinguish high- from low-reliability predictions. We integrate CoVar as a plug-in module into representative semi-supervised semantic segmentation and image classification methods. Across PASCAL VOC 2012, Cityscapes, CIFAR-10, and Mini-ImageNet with varying label ratios and backbones, it consistently improves over strong baselines, indicating that combining confidence with residual-class variance provides a more reliable basis for pseudo-label selection than fixed confidence thresholds. (Code: https://github.com/ljs11528/CoVar_Pseudo_Label_Selection.git)