Comparing effectiveness of regularization methods on text classification: Simple and complex model in data shortage situation

TL;DR

This study evaluates how different regularization techniques impact the performance of simple and complex text classification models in data-scarce scenarios, demonstrating that regularization improves results especially for complex models.

Contribution

It compares regularization effects on simple and complex models using various methods and datasets under extreme data shortage conditions.

Findings

Regularization improves model performance in low-data regimes.

Complex models benefit more from adversarial and semi-supervised regularization.

Simple models are inherently more robust to overfitting.

Abstract

Text classification is the task of assigning a document to a predefined class. However, it is expensive to acquire enough labeled documents or to label them. In this paper, we study the regularization methods' effects on various classification models when only a few labeled data are available. We compare a simple word embedding-based model, which is simple but effective, with complex models (CNN and BiLSTM). In supervised learning, adversarial training can further regularize the model. When an unlabeled dataset is available, we can regularize the model using semi-supervised learning methods such as the Pi model and virtual adversarial training. We evaluate the regularization effects on four text classification datasets (AG news, DBpedia, Yahoo! Answers, Yelp Polarity), using only 0.1% to 0.5% of the original labeled training documents. The simple model performs relatively well in fully supervised learning, but with the help of adversarial training and semi-supervised learning, both simple and complex models can be regularized, showing better results for complex models. Although the simple model is robust to overfitting, a complex model with well-designed prior beliefs can be also robust to overfitting.