ZeroBERTo: Leveraging Zero-Shot Text Classification by Topic Modeling

TL;DR

ZeroBERTo introduces an unsupervised clustering approach to improve zero-shot text classification, especially for long texts, achieving better performance and efficiency over transformer-based models.

Contribution

The paper presents ZeroBERTo, a novel zero-shot classification model that uses topic modeling for data compression, addressing long text handling and execution time issues.

Findings

ZeroBERTo outperforms XLM-R by 12% in F1 score on FolhaUOL dataset.

ZeroBERTo is more efficient with shorter execution times.

The model handles long inputs better than transformer-based approaches.

Abstract

Traditional text classification approaches often require a good amount of labeled data, which is difficult to obtain, especially in restricted domains or less widespread languages. This lack of labeled data has led to the rise of low-resource methods, that assume low data availability in natural language processing. Among them, zero-shot learning stands out, which consists of learning a classifier without any previously labeled data. The best results reported with this approach use language models such as Transformers, but fall into two problems: high execution time and inability to handle long texts as input. This paper proposes a new model, ZeroBERTo, which leverages an unsupervised clustering step to obtain a compressed data representation before the classification task. We show that ZeroBERTo has better performance for long inputs and shorter execution time, outperforming XLM-R by about 12% in the F1 score in the FolhaUOL dataset. Keywords: Low-Resource NLP, Unlabeled data, Zero-Shot Learning, Topic Modeling, Transformers.