Informative missingness and its implications in semi-supervised learning

TL;DR

This paper explores how modeling the mechanism behind missing labels in semi-supervised learning can improve classifier performance, especially when missingness is informative and data are sparse or overlapping.

Contribution

It introduces a statistical framework for incorporating informative missingness in SSL, demonstrating potential performance gains over traditional fully labelled approaches.

Findings

Modeling missing-label mechanisms can outperform fully labelled data in certain conditions.

Informative missingness reduces expected classification error.

The framework unifies likelihood inference with empirical SSL methods.

Abstract

Semi-supervised learning (SSL) constructs classifiers using both labelled and unlabelled data. It leverages information from labelled samples, whose acquisition is often costly or labour-intensive, together with unlabelled data to enhance prediction performance. This defines an incomplete-data problem, which statistically can be formulated within the likelihood framework for finite mixture models that can be fitted using the expectation-maximisation (EM) algorithm. Ideally, one would prefer a completely labelled sample, as one would anticipate that a labelled observation provides more information than an unlabelled one. However, when the mechanism governing label absence depends on the observed features or the class labels or both, the missingness indicators themselves contain useful information. In certain situations, the information gained from modelling the missing-label mechanism can even outweigh the loss due to missing labels, yielding a classifier with a smaller expected error than one based on a completely labelled sample analysed. This improvement arises particularly when class overlap is moderate, labelled data are sparse, and the missingness is informative. Modelling such informative missingness thus offers a coherent statistical framework that unifies likelihood-based inference with the behaviour of empirical SSL methods.