MarginMatch: Improving Semi-Supervised Learning with Pseudo-Margins

TL;DR

MarginMatch is a semi-supervised learning method that enhances pseudo-label quality by analyzing training dynamics, leading to significant improvements on vision benchmarks with limited labeled data.

Contribution

It introduces a novel pseudo-labeling approach that uses training dynamics to improve pseudo-label quality in semi-supervised learning.

Findings

3.25% error rate reduction on CIFAR-100 with 25 labels per class

3.78% error rate reduction on STL-10 with 4 labels per class

Substantial improvements on multiple vision benchmarks

Abstract

We introduce MarginMatch, a new SSL approach combining consistency regularization and pseudo-labeling, with its main novelty arising from the use of unlabeled data training dynamics to measure pseudo-label quality. Instead of using only the model's confidence on an unlabeled example at an arbitrary iteration to decide if the example should be masked or not, MarginMatch also analyzes the behavior of the model on the pseudo-labeled examples as the training progresses, to ensure low quality predictions are masked out. MarginMatch brings substantial improvements on four vision benchmarks in low data regimes and on two large-scale datasets, emphasizing the importance of enforcing high-quality pseudo-labels. Notably, we obtain an improvement in error rate over the state-of-the-art of 3.25% on CIFAR-100 with only 25 labels per class and of 3.78% on STL-10 using as few as 4 labels per class. We make our code available at https://github.com/tsosea2/MarginMatch.