KD-FixMatch: Knowledge Distillation Siamese Neural Networks

TL;DR

KD-FixMatch introduces a knowledge distillation approach to semi-supervised learning, improving upon FixMatch by reducing performance degradation from noisy pseudo labels through sequential and simultaneous training of siamese neural networks.

Contribution

The paper proposes KD-FixMatch, a novel SSL algorithm that enhances FixMatch with knowledge distillation and a two-stage training process for better performance with noisy pseudo labels.

Findings

KD-FixMatch outperforms FixMatch on four datasets.

The method achieves higher accuracy with noisy pseudo labels.

Sequential training improves the quality of pseudo labels.

Abstract

Semi-supervised learning (SSL) has become a crucial approach in deep learning as a way to address the challenge of limited labeled data. The success of deep neural networks heavily relies on the availability of large-scale high-quality labeled data. However, the process of data labeling is time-consuming and unscalable, leading to shortages in labeled data. SSL aims to tackle this problem by leveraging additional unlabeled data in the training process. One of the popular SSL algorithms, FixMatch, trains identical weight-sharing teacher and student networks simultaneously using a siamese neural network (SNN). However, it is prone to performance degradation when the pseudo labels are heavily noisy in the early training stage. We present KD-FixMatch, a novel SSL algorithm that addresses the limitations of FixMatch by incorporating knowledge distillation. The algorithm utilizes a combination of sequential and simultaneous training of SNNs to enhance performance and reduce performance degradation. Firstly, an outer SNN is trained using labeled and unlabeled data. After that, the network of the well-trained outer SNN generates pseudo labels for the unlabeled data, from which a subset of unlabeled data with trusted pseudo labels is then carefully created through high-confidence sampling and deep embedding clustering. Finally, an inner SNN is trained with the labeled data, the unlabeled data, and the subset of unlabeled data with trusted pseudo labels. Experiments on four public data sets demonstrate that KD-FixMatch outperforms FixMatch in all cases. Our results indicate that KD-FixMatch has a better training starting point that leads to improved model performance compared to FixMatch.