Exploiting All Samples in Low-Resource Sentence Classification: Early Stopping and Initialization Parameters

TL;DR

This paper investigates how to best utilize limited labeled data in low-resource sentence classification by exploring training strategies, proposing an integrated approach, and demonstrating its effectiveness across multiple datasets and models.

Contribution

It introduces an integrated method combining weight averaging and non-validation stopping, improving performance without additional data or model redesigns in low-resource settings.

Findings

The integrated method outperforms conventional validation-based methods by 1.8% accuracy on average.

Performance varies significantly with different training strategies and approaches.

The method enhances state-of-the-art models using additional data or redesigns.

Abstract

To improve deep-learning performance in low-resource settings, many researchers have redesigned model architectures or applied additional data (e.g., external resources, unlabeled samples). However, there have been relatively few discussions on how to make good use of small amounts of labeled samples, although it is potentially beneficial and should be done before applying additional data or redesigning models. In this study, we assume a low-resource setting in which only a few labeled samples (i.e., 30-100 per class) are available, and we discuss how to exploit them without additional data or model redesigns. We explore possible approaches in the following three aspects: training-validation splitting, early stopping, and weight initialization. Extensive experiments are conducted on six public sentence classification datasets. Performance on various evaluation metrics (e.g., accuracy, loss, and calibration error) significantly varied depending on the approaches that were combined in the three aspects. Based on the results, we propose an integrated method, which is to initialize the model with a weight averaging method and use a non-validation stop method to train all samples. This simple integrated method consistently outperforms the competitive methods; e.g., the average accuracy of six datasets of this method was 1.8% higher than those of conventional validation-based methods. In addition, the integrated method further improves the performance when adapted to several state-of-the-art models that use additional data or redesign the network architecture (e.g., self-training and enhanced structural models). Our results highlight the importance of the training strategy and suggest that the integrated method can be the first step in the low-resource setting. This study provides empirical knowledge that will be helpful when dealing with low-resource data in future efforts.