Binary Classification with Confidence Difference

TL;DR

This paper introduces a novel weakly supervised binary classification method called confidence-difference classification, which uses unlabeled data pairs with confidence difference instead of pointwise confidence labels, achieving optimal convergence and reducing overfitting.

Contribution

It proposes a risk-consistent approach and a risk correction method for confidence-difference classification, with proven theoretical guarantees and validated experimental results.

Findings

Achieves optimal convergence rate in estimation error bounds.

Effectively mitigates overfitting through risk correction.

Demonstrates superior performance on benchmark and real-world datasets.

Abstract

Recently, learning with soft labels has been shown to achieve better performance than learning with hard labels in terms of model generalization, calibration, and robustness. However, collecting pointwise labeling confidence for all training examples can be challenging and time-consuming in real-world scenarios. This paper delves into a novel weakly supervised binary classification problem called confidence-difference (ConfDiff) classification. Instead of pointwise labeling confidence, we are given only unlabeled data pairs with confidence difference that specifies the difference in the probabilities of being positive. We propose a risk-consistent approach to tackle this problem and show that the estimation error bound achieves the optimal convergence rate. We also introduce a risk correction approach to mitigate overfitting problems, whose consistency and convergence rate are also proven. Extensive experiments on benchmark data sets and a real-world recommender system data set validate the effectiveness of our proposed approaches in exploiting the supervision information of the confidence difference.