RooseBERT: A New Deal For Political Language Modelling

TL;DR

RooseBERT is a domain-specific language model trained on large political debate corpora, significantly improving performance on various political discourse analysis tasks compared to general-purpose models.

Contribution

We introduce RooseBERT, a novel pre-trained language model tailored for political language, trained on large-scale political debate data, and demonstrate its superior performance on multiple political analysis tasks.

Findings

RooseBERT outperforms general-purpose LMs on political stance detection.

It achieves higher accuracy in argument component classification.

It significantly improves sentiment analysis in political contexts.

Abstract

The increasing amount of political debates and politics-related discussions calls for the definition of novel computational methods to automatically analyse such content with the final goal of lightening up political deliberation to citizens. However, the specificity of the political language and the argumentative form of these debates (employing hidden communication strategies and leveraging implicit arguments) make this task very challenging, even for current general-purpose pre-trained Language Models (LMs). To address this, we introduce a novel pre-trained LM for political discourse language called RooseBERT. Pre-training a LM on a specialised domain presents different technical and linguistic challenges, requiring extensive computational resources and large-scale data. RooseBERT has been trained on large political debate and speech corpora (11GB) in English. To evaluate its performances, we fine-tuned it on multiple downstream tasks related to political debate analysis, i.e., stance detection, sentiment analysis, argument component detection and classification, argument relation prediction and classification, policy classification, named entity recognition (NER). Our results show significant improvements over general-purpose LMs on the majority of these tasks, highlighting how domain-specific pre-training enhances performance in political debate analysis. We release RooseBERT for the research community.