Certainty Driven Consistency Loss on Multi-Teacher Networks for Semi-Supervised Learning

TL;DR

This paper introduces a certainty-driven consistency loss for semi-supervised learning that uses predictive uncertainty to improve the quality of teacher targets, leading to better student model training.

Contribution

The paper proposes a novel uncertainty-aware consistency loss with filtering and temperature approaches, and a decoupled framework to enhance semi-supervised learning performance.

Findings

Outperforms existing methods on SVHN, CIFAR-10, and CIFAR-100 datasets.

Effectively filters out uncertain predictions to improve learning.

Enhances model robustness through a decoupled training framework.

Abstract

One of the successful approaches in semi-supervised learning is based on the consistency regularization. Typically, a student model is trained to be consistent with teacher prediction for the inputs under different perturbations. To be successful, the prediction targets given by teacher should have good quality, otherwise the student can be misled by teacher. Unfortunately, existing methods do not assess the quality of the teacher targets. In this paper, we propose a novel Certainty-driven Consistency Loss (CCL) that exploits the predictive uncertainty in the consistency loss to let the student dynamically learn from reliable targets. Specifically, we propose two approaches, i.e. Filtering CCL and Temperature CCL to either filter out uncertain predictions or pay less attention on them in the consistency regularization. We further introduce a novel decoupled framework to encourage model difference. Experimental results on SVHN, CIFAR-10, and CIFAR-100 demonstrate the advantages of our method over a few existing methods.