Overlapping Word Removal is All You Need: Revisiting Data Imbalance in Hope Speech Detection

TL;DR

This paper investigates data imbalance in hope speech detection and demonstrates that simple pre-processing, along with focal loss and data augmentation, significantly improves model performance using multilingual BERT.

Contribution

It introduces overlapping word removal as a simple yet effective pre-processing step and evaluates its impact alongside focal loss and data augmentation for hope speech detection.

Findings

Overlapping word removal improves F1-Macro by 0.28

Focal loss increases F1-Macro by 0.11

Data augmentation with back-translation improves results by 0.10

Abstract

Hope Speech Detection, a task of recognizing positive expressions, has made significant strides recently. However, much of the current works focus on model development without considering the issue of inherent imbalance in the data. Our work revisits this issue in hope-speech detection by introducing focal loss, data augmentation, and pre-processing strategies. Accordingly, we find that introducing focal loss as part of Multilingual-BERT's (M-BERT) training process mitigates the effect of class imbalance and improves overall F1-Macro by 0.11. At the same time, contextual and back-translation-based word augmentation with M-BERT improves results by 0.10 over baseline despite imbalance. Finally, we show that overlapping word removal based on pre-processing, though simple, improves F1-Macro by 0.28. In due process, we present detailed studies depicting various behaviors of each of these strategies and summarize key findings from our empirical results for those interested in getting the most out of M-BERT for hope speech detection under real-world conditions of data imbalance.