Robust Training of Federated Models with Extremely Label Deficiency

TL;DR

This paper introduces Twin-sight, a twin-model paradigm for federated semi-supervised learning that mitigates gradient conflicts and improves model performance under label deficiency.

Contribution

It proposes a novel twin-model approach with a neighborhood-preserving constraint to enhance mutual guidance in federated semi-supervised learning.

Findings

Twin-sight significantly outperforms state-of-the-art methods.

The neighborhood-preserving constraint improves model synergy.

Experimental results validate the effectiveness of Twin-sight.

Abstract

Federated semi-supervised learning (FSSL) has emerged as a powerful paradigm for collaboratively training machine learning models using distributed data with label deficiency. Advanced FSSL methods predominantly focus on training a single model on each client. However, this approach could lead to a discrepancy between the objective functions of labeled and unlabeled data, resulting in gradient conflicts. To alleviate gradient conflict, we propose a novel twin-model paradigm, called Twin-sight, designed to enhance mutual guidance by providing insights from different perspectives of labeled and unlabeled data. In particular, Twin-sight concurrently trains a supervised model with a supervised objective function while training an unsupervised model using an unsupervised objective function. To enhance the synergy between these two models, Twin-sight introduces a neighbourhood-preserving constraint, which encourages the preservation of the neighbourhood relationship among data features extracted by both models. Our comprehensive experiments on four benchmark datasets provide substantial evidence that Twin-sight can significantly outperform state-of-the-art methods across various experimental settings, demonstrating the efficacy of the proposed Twin-sight.