A Dual-Layered Evaluation of Geopolitical and Cultural Bias in LLMs

TL;DR

This paper introduces a dual-phase framework to evaluate geopolitical and cultural biases in large language models across factual and disputable questions in multiple languages, highlighting how query language influences model responses.

Contribution

It presents a novel two-phase evaluation method distinguishing model bias from inference bias, with a curated multilingual dataset for assessing LLM behavior on sensitive topics.

Findings

Phase 1 shows query language alignment in factual questions.

Phase 2 reveals interplay between training context and query language.

Framework aids culturally aware evaluation of multilingual LLMs.

Abstract

As large language models (LLMs) are increasingly deployed across diverse linguistic and cultural contexts, understanding their behavior in both factual and disputable scenarios is essential, especially when their outputs may shape public opinion or reinforce dominant narratives. In this paper, we define two types of bias in LLMs: model bias (bias stemming from model training) and inference bias (bias induced by the language of the query), through a two-phase evaluation. Phase 1 evaluates LLMs on factual questions where a single verifiable answer exists, assessing whether models maintain consistency across different query languages. Phase 2 expands the scope by probing geopolitically sensitive disputes, where responses may reflect culturally embedded or ideologically aligned perspectives. We construct a manually curated dataset spanning both factual and disputable QA, across four languages and question types. The results show that Phase 1 exhibits query language induced alignment, while Phase 2 reflects an interplay between the model's training context and query language. This paper offers a structured framework for evaluating LLM behavior across neutral and sensitive topics, providing insights for future LLM deployment and culturally aware evaluation practices in multilingual contexts.