WC-SBERT: Zero-Shot Text Classification via SBERT with Self-Training for Wikipedia Categories

TL;DR

This paper introduces WC-SBERT, a zero-shot text classification method using SBERT with a novel self-training strategy that leverages Wikipedia categories, significantly reducing training time and achieving state-of-the-art results.

Contribution

The paper presents a new self-training approach that trains on labels rather than text, enabling rapid adaptation and improved efficiency in zero-shot classification.

Findings

Achieves state-of-the-art results on Yahoo Topic and AG News datasets.

Reduces training time to minutes by using Wikipedia categories as a unified training set.

Significantly decreases training data requirements compared to traditional methods.

Abstract

Our research focuses on solving the zero-shot text classification problem in NLP, with a particular emphasis on innovative self-training strategies. To achieve this objective, we propose a novel self-training strategy that uses labels rather than text for training, significantly reducing the model's training time. Specifically, we use categories from Wikipedia as our training set and leverage the SBERT pre-trained model to establish positive correlations between pairs of categories within the same text, facilitating associative training. For new test datasets, we have improved the original self-training approach, eliminating the need for prior training and testing data from each target dataset. Instead, we adopt Wikipedia as a unified training dataset to better approximate the zero-shot scenario. This modification allows for rapid fine-tuning and inference across different datasets, greatly reducing the time required for self-training. Our experimental results demonstrate that this method can adapt the model to the target dataset within minutes. Compared to other BERT-based transformer models, our approach significantly reduces the amount of training data by training only on labels, not the actual text, and greatly improves training efficiency by utilizing a unified training set. Additionally, our method achieves state-of-the-art results on both the Yahoo Topic and AG News datasets.