Task-Adaptive Pre-Training for Boosting Learning With Noisy Labels: A Study on Text Classification for African Languages

TL;DR

This paper investigates the effectiveness of task-adaptive pre-training techniques in improving text classification performance in low-resource African languages with noisy labels, comparing various noise-handling methods.

Contribution

It demonstrates that task-adaptive pre-training significantly enhances learning with noisy labels in low-resource language text classification tasks.

Findings

Task-adaptive pre-training improves accuracy with noisy labels.

Noise-handling methods combined with pre-training yield better results.

Realistic noise scenarios are effectively mitigated by the proposed approach.

Abstract

For high-resource languages like English, text classification is a well-studied task. The performance of modern NLP models easily achieves an accuracy of more than 90% in many standard datasets for text classification in English (Xie et al., 2019; Yang et al., 2019; Zaheer et al., 2020). However, text classification in low-resource languages is still challenging due to the lack of annotated data. Although methods like weak supervision and crowdsourcing can help ease the annotation bottleneck, the annotations obtained by these methods contain label noise. Models trained with label noise may not generalize well. To this end, a variety of noise-handling techniques have been proposed to alleviate the negative impact caused by the errors in the annotations (for extensive surveys see (Hedderich et al., 2021; Algan & Ulusoy, 2021)). In this work, we experiment with a group of standard noisy-handling methods on text classification tasks with noisy labels. We study both simulated noise and realistic noise induced by weak supervision. Moreover, we find task-adaptive pre-training techniques (Gururangan et al., 2020) are beneficial for learning with noisy labels.