Don't fear the unlabelled: safe semi-supervised learning via simple debiasing

TL;DR

This paper introduces a simple debiasing technique for semi-supervised learning that removes bias from risk estimates, providing theoretical guarantees and improving calibration of existing SSL methods like Pseudo-label and Fixmatch.

Contribution

The authors propose a straightforward debiasing approach applicable to most deep SSL methods, with theoretical generalisation bounds and improved empirical performance.

Findings

Debiasing improves model calibration in SSL.

The method provides theoretical generalisation error bounds.

Debiased SSL methods outperform or match traditional techniques.

Abstract

Semi-supervised learning (SSL) provides an effective means of leveraging unlabelled data to improve a model performance. Even though the domain has received a considerable amount of attention in the past years, most methods present the common drawback of lacking theoretical guarantees. Our starting point is to notice that the estimate of the risk that most discriminative SSL methods minimise is biased, even asymptotically. This bias impedes the use of standard statistical learning theory and can hurt empirical performance. We propose a simple way of removing the bias. Our debiasing approach is straightforward to implement and applicable to most deep SSL methods. We provide simple theoretical guarantees on the trustworthiness of these modified methods, without having to rely on the strong assumptions on the data distribution that SSL theory usually requires. In particular, we provide generalisation error bounds for the proposed methods. We evaluate debiased versions of different existing SSL methods, such as the Pseudo-label method and Fixmatch, and show that debiasing can compete with classic deep SSL techniques in various settings by providing better calibrated models. Additionally, we provide a theoretical explanation of the intuition of the popular SSL methods.