Semi-supervised learning objectives as log-likelihoods in a generative model of data curation

TL;DR

This paper presents a novel theoretical framework that interprets semi-supervised learning objectives as log-likelihoods within a generative model of data curation, providing insights into why SSL performs better on curated datasets.

Contribution

It formulates SSL objectives as log-likelihoods in a generative model, enabling Bayesian SSL development and explaining SSL's effectiveness on curated data.

Findings

SSL methods are lower bounds on a principled log-likelihood.

Bayesian SSL demonstrated on toy data supports the theory.

SSL performs significantly better on curated datasets than uncurated ones.

Abstract

We currently do not have an understanding of semi-supervised learning (SSL) objectives such as pseudo-labelling and entropy minimization as log-likelihoods, which precludes the development of e.g. Bayesian SSL. Here, we note that benchmark image datasets such as CIFAR-10 are carefully curated, and we formulate SSL objectives as a log-likelihood in a generative model of data curation that was initially developed to explain the cold-posterior effect (Aitchison 2020). SSL methods, from entropy minimization and pseudo-labelling, to state-of-the-art techniques similar to FixMatch can be understood as lower-bounds on our principled log-likelihood. We are thus able to give a proof-of-principle for Bayesian SSL on toy data. Finally, our theory suggests that SSL is effective in part due to the statistical patterns induced by data curation. This provides an explanation of past results which show SSL performs better on clean datasets without any "out of distribution" examples. Confirming these results we find that SSL gave much larger performance improvements on curated than on uncurated data, using matched curated and uncurated datasets based on Galaxy Zoo 2.