Task Adaptive Pretraining of Transformers for Hostility Detection

TL;DR

This paper demonstrates that Task Adaptive Pretraining (TAPT) significantly improves Transformer-based models for detecting hostility in social media, achieving top results in Hindi tweet classification tasks.

Contribution

It introduces the application of TAPT prior to fine-tuning Transformer models for hostility detection, showing performance gains in both coarse and fine-grained classification tasks.

Findings

TAPT enhances classification accuracy for hostility detection.

The system achieved 97.16% F1 score in coarse classification.

It ranked first in the shared task for Hindi hostility detection.

Abstract

Identifying adverse and hostile content on the web and more particularly, on social media, has become a problem of paramount interest in recent years. With their ever increasing popularity, fine-tuning of pretrained Transformer-based encoder models with a classifier head are gradually becoming the new baseline for natural language classification tasks. In our work, we explore the gains attributed to Task Adaptive Pretraining (TAPT) prior to fine-tuning of Transformer-based architectures. We specifically study two problems, namely, (a) Coarse binary classification of Hindi Tweets into Hostile or Not, and (b) Fine-grained multi-label classification of Tweets into four categories: hate, fake, offensive, and defamation. Building up on an architecture which takes emojis and segmented hashtags into consideration for classification, we are able to experimentally showcase the performance upgrades due to TAPT. Our system (with team name 'iREL IIIT') ranked first in the 'Hostile Post Detection in Hindi' shared task with an F1 score of 97.16% for coarse-grained detection and a weighted F1 score of 62.96% for fine-grained multi-label classification on the provided blind test corpora.