Political Leaning and Politicalness Classification of Texts

TL;DR

This paper evaluates transformer-based models for classifying texts by political leaning and politicalness, highlighting limitations of current approaches and proposing a diverse dataset for improved generalization.

Contribution

It compiles a comprehensive, diverse dataset and benchmarks models to improve out-of-distribution performance in political text classification.

Findings

Current models perform poorly on out-of-distribution texts

A new diverse dataset enhances model generalization

Benchmarking reveals strengths and weaknesses of existing approaches

Abstract

This paper addresses the challenge of automatically classifying text according to political leaning and politicalness using transformer models. We compose a comprehensive overview of existing datasets and models for these tasks, finding that current approaches create siloed solutions that perform poorly on out-of-distribution texts. To address this limitation, we compile a diverse dataset by combining 12 datasets for political leaning classification and creating a new dataset for politicalness by extending 18 existing datasets with the appropriate label. Through extensive benchmarking with leave-one-in and leave-one-out methodologies, we evaluate the performance of existing models and train new ones with enhanced generalization capabilities.