Smooth Neighbors on Teacher Graphs for Semi-supervised Learning

TL;DR

The paper introduces SNTG, a semi-supervised learning method that leverages teacher graph-based smoothness constraints to improve model performance, especially with fewer labels and noisy data.

Contribution

It proposes a novel graph-based regularization method that enforces smoothness on the teacher's prediction graph, enhancing semi-supervised learning performance.

Findings

Achieved state-of-the-art results on CIFAR-10 and SVHN benchmarks.

Significant error rate reductions on MNIST with very few labels.

Demonstrated robustness to noisy labels.

Abstract

The recently proposed self-ensembling methods have achieved promising results in deep semi-supervised learning, which penalize inconsistent predictions of unlabeled data under different perturbations. However, they only consider adding perturbations to each single data point, while ignoring the connections between data samples. In this paper, we propose a novel method, called Smooth Neighbors on Teacher Graphs (SNTG). In SNTG, a graph is constructed based on the predictions of the teacher model, i.e., the implicit self-ensemble of models. Then the graph serves as a similarity measure with respect to which the representations of "similar" neighboring points are learned to be smooth on the low-dimensional manifold. We achieve state-of-the-art results on semi-supervised learning benchmarks. The error rates are 9.89%, 3.99% for CIFAR-10 with 4000 labels, SVHN with 500 labels, respectively. In particular, the improvements are significant when the labels are fewer. For the non-augmented MNIST with only 20 labels, the error rate is reduced from previous 4.81% to 1.36%. Our method also shows robustness to noisy labels.