Lightweight Contenders: Navigating Semi-Supervised Text Mining through Peer Collaboration and Self Transcendence

TL;DR

This paper presents PS-NET, a novel semi-supervised learning framework for lightweight text mining models that leverages peer collaboration, self-augmentation, and online distillation to improve performance with scarce labeled data.

Contribution

Introduction of PS-NET, a framework combining online distillation, peer ensemble collaboration, and adversarial self-augmentation for semi-supervised lightweight text classification.

Findings

PS-NET outperforms state-of-the-art lightweight SSL models.

Uses a 2-layer distilled BERT for efficiency.

Achieves significant improvements with very limited labeled data.

Abstract

The semi-supervised learning (SSL) strategy in lightweight models requires reducing annotated samples and facilitating cost-effective inference. However, the constraint on model parameters, imposed by the scarcity of training labels, limits the SSL performance. In this paper, we introduce PS-NET, a novel framework tailored for semi-supervised text mining with lightweight models. PS-NET incorporates online distillation to train lightweight student models by imitating the Teacher model. It also integrates an ensemble of student peers that collaboratively instruct each other. Additionally, PS-NET implements a constant adversarial perturbation schema to further self-augmentation by progressive generalizing. Our PS-NET, equipped with a 2-layer distilled BERT, exhibits notable performance enhancements over SOTA lightweight SSL frameworks of FLiText and DisCo in SSL text classification with extremely rare labelled data.