Classifying multilingual party manifestos: Domain transfer across country, time, and genre

TL;DR

This study evaluates the effectiveness of domain transfer using transformer models for classifying multilingual political manifestos across different countries, genres, and time periods, highlighting the models' robustness and transferability.

Contribution

It demonstrates the strong within-domain performance of fine-tuned transformer models and assesses their robustness across various dimensions such as country, language, genre, and time.

Findings

BERT achieves the highest initial classification scores.

DistilBERT offers a competitive, computationally efficient alternative.

Models can be applied to future data with similar performance.

Abstract

Annotating costs of large corpora are still one of the main bottlenecks in empirical social science research. On the one hand, making use of the capabilities of domain transfer allows re-using annotated data sets and trained models. On the other hand, it is not clear how well domain transfer works and how reliable the results are for transfer across different dimensions. We explore the potential of domain transfer across geographical locations, languages, time, and genre in a large-scale database of political manifestos. First, we show the strong within-domain classification performance of fine-tuned transformer models. Second, we vary the genre of the test set across the aforementioned dimensions to test for the fine-tuned models' robustness and transferability. For switching genres, we use an external corpus of transcribed speeches from New Zealand politicians while for the other three dimensions, custom splits of the Manifesto database are used. While BERT achieves the best scores in the initial experiments across modalities, DistilBERT proves to be competitive at a lower computational expense and is thus used for further experiments across time and country. The results of the additional analysis show that (Distil)BERT can be applied to future data with similar performance. Moreover, we observe (partly) notable differences between the political manifestos of different countries of origin, even if these countries share a language or a cultural background.