FLiText: A Faster and Lighter Semi-Supervised Text Classification with Convolution Networks

TL;DR

FLiText is a novel semi-supervised learning framework that significantly improves lightweight text classification models' accuracy with minimal labeled data, outperforming existing methods on multiple benchmarks.

Contribution

Introduces FLiText, a semi-supervised framework with an inspirer network and consistency regularization, tailored for lightweight NLP models, achieving state-of-the-art results.

Findings

FLiText improves TextCNN accuracy from 51.00% to 90.49% on IMDb.

FLiText enhances Yelp-5 accuracy from 39.8% to 58.06%.

FLiText achieves over 6% accuracy gain using less than 1% labeled data.

Abstract

In natural language processing (NLP), state-of-the-art (SOTA) semi-supervised learning (SSL) frameworks have shown great performance on deep pre-trained language models such as BERT, and are expected to significantly reduce the demand for manual labeling. However, our empirical studies indicate that these frameworks are not suitable for lightweight models such as TextCNN, LSTM and etc. In this work, we develop a new SSL framework called FLiText, which stands for Faster and Lighter semi-supervised Text classification. FLiText introduces an inspirer network together with the consistency regularization framework, which leverages a generalized regular constraint on the lightweight models for efficient SSL. As a result, FLiText obtains new SOTA performance for lightweight models across multiple SSL benchmarks on text classification. Compared with existing SOTA SSL methods on TextCNN, FLiText improves the accuracy of lightweight model TextCNN from 51.00% to 90.49% on IMDb, 39.8% to 58.06% on Yelp-5, and from 55.3% to 65.08% on Yahoo. In addition, compared with the fully supervised method on the full dataset, FLiText just uses less than 1% of labeled data to improve the accuracy by 6.59%, 3.94%, and 3.22% on the datasets of IMDb, Yelp-5, and Yahoo respectively.