Transfer and Active Learning for Dissonance Detection: Addressing the Rare-Class Challenge

TL;DR

This paper explores transfer and active learning methods, including a new probability-of-rare-class strategy, to improve dissonance detection in social media, addressing challenges of rare class data scarcity.

Contribution

It introduces and evaluates transfer- and active learning strategies, notably the PRC approach, for rare class dissonance detection, highlighting their effectiveness and limitations.

Findings

PRC strategy effectively guides annotations and improves accuracy

Transfer learning enhances cold-start performance but not active learning iterations

Iterative transfer learning does not benefit from multiple active learning cycles

Abstract

While transformer-based systems have enabled greater accuracies with fewer training examples, data acquisition obstacles still persist for rare-class tasks -- when the class label is very infrequent (e.g. < 5% of samples). Active learning has in general been proposed to alleviate such challenges, but choice of selection strategy, the criteria by which rare-class examples are chosen, has not been systematically evaluated. Further, transformers enable iterative transfer-learning approaches. We propose and investigate transfer- and active learning solutions to the rare class problem of dissonance detection through utilizing models trained on closely related tasks and the evaluation of acquisition strategies, including a proposed probability-of-rare-class (PRC) approach. We perform these experiments for a specific rare class problem: collecting language samples of cognitive dissonance from social media. We find that PRC is a simple and effective strategy to guide annotations and ultimately improve model accuracy while transfer-learning in a specific order can improve the cold-start performance of the learner but does not benefit iterations of active learning.