Tradeoffs in Resampling and Filtering for Imbalanced Classification

TL;DR

This paper investigates the tradeoffs in resampling and filtering methods for imbalanced token classification, highlighting how data selection impacts model performance and how base rates influence these effects.

Contribution

It provides an empirical analysis of how different data selection strategies affect performance in imbalanced NLP tasks, emphasizing the importance of filtering test data.

Findings

Filtering test data is as crucial as training data selection in highly imbalanced scenarios.

The base rate of the positive class affects the magnitude of performance tradeoffs.

Different data selection methods trade effectiveness for efficiency.

Abstract

Imbalanced classification problems are extremely common in natural language processing and are solved using a variety of resampling and filtering techniques, which often involve making decisions on how to select training data or decide which test examples should be labeled by the model. We examine the tradeoffs in model performance involved in choices of training sample and filter training and test data in heavily imbalanced token classification task and examine the relationship between the magnitude of these tradeoffs and the base rate of the phenomenon of interest. In experiments on sequence tagging to detect rare phenomena in English and Arabic texts, we find that different methods of selecting training data bring tradeoffs in effectiveness and efficiency. We also see that in highly imbalanced cases, filtering test data using first-pass retrieval models is as important for model performance as selecting training data. The base rate of a rare positive class has a clear effect on the magnitude of the changes in performance caused by the selection of training or test data. As the base rate increases, the differences brought about by those choices decreases.