Framing Political Bias in Multilingual LLMs Across Pakistani Languages

TL;DR

This paper systematically evaluates political bias in 13 multilingual LLMs across five Pakistani languages, revealing language-conditioned ideological patterns and emphasizing the importance of culturally grounded bias assessment.

Contribution

It introduces a culturally adapted Political Compass Test framework for multilingual bias evaluation in Pakistani languages, highlighting language-specific ideological biases in state-of-the-art LLMs.

Findings

LLMs show liberal-left bias similar to Western data

Regional languages exhibit more authoritarian framing

Model-specific bias patterns are consistent across languages

Abstract

Large Language Models (LLMs) increasingly shape public discourse, yet most evaluations of political and economic bias have focused on high-resource, Western languages and contexts. This leaves critical blind spots in low-resource, multilingual regions such as Pakistan, where linguistic identity is closely tied to political, religious, and regional ideologies. We present a systematic evaluation of political bias in 13 state-of-the-art LLMs across five Pakistani languages: Urdu, Punjabi, Sindhi, Pashto, and Balochi. Our framework integrates a culturally adapted Political Compass Test (PCT) with multi-level framing analysis, capturing both ideological stance (economic/social axes) and stylistic framing (content, tone, emphasis). Prompts are aligned with 11 socio-political themes specific to the Pakistani context. Results show that while LLMs predominantly reflect liberal-left orientations consistent with Western training data, they exhibit more authoritarian framing in regional languages, highlighting language-conditioned ideological modulation. We also identify consistent model-specific bias patterns across languages. These findings show the need for culturally grounded, multilingual bias auditing frameworks in global NLP.