Assessing the Political Fairness of Multilingual LLMs: A Case Study based on a 21-way Multiparallel EuroParl Dataset

TL;DR

This paper introduces a multilingual translation fairness framework to assess political biases in LLMs, using a new EuroParl dataset that reveals systematic translation quality differences linked to political affiliations.

Contribution

It presents a novel multilingual fairness approach for evaluating political biases in LLMs and introduces a comprehensive 21-way EuroParl dataset with political labels.

Findings

Majority parties are better translated than outsider parties.

Systematic translation quality differences correlate with political affiliations.

New dataset enables large-scale multilingual political bias analysis.

Abstract

The political biases of Large Language Models (LLMs) are usually assessed by simulating their answers to English surveys. In this work, we propose an alternative framing of political biases, relying on principles of fairness in multilingual translation. We systematically compare the translation quality of speeches in the European Parliament (EP), observing systematic differences with majority parties from left and right being better translated than outsider parties. This study is made possible by a new, 21-way multiparallel version of EuroParl, the parliamentary proceedings of the EP, which includes the political affiliations of each speaker. The dataset consists of 1.5M sentences for a total of 40M words and 249M characters. It covers three years, 1000+ speakers, 7 countries, 12 EU parties, 25 EU committees, and hundreds of national parties.