Regional Bias in Large Language Models

TL;DR

This paper evaluates regional bias in ten large language models using a new prompt-based framework, revealing significant variation in bias levels that impact fairness and inclusivity in AI applications.

Contribution

It introduces FAZE, a novel evaluation framework for measuring regional bias in LLMs, and provides systematic analysis across multiple models highlighting bias disparities.

Findings

GPT-3.5 has the highest regional bias score (9.5)

Claude 3.5 Sonnet exhibits the lowest bias score (2.5)

Bias levels vary significantly across models, affecting fairness.

Abstract

This study investigates regional bias in large language models (LLMs), an emerging concern in AI fairness and global representation. We evaluate ten prominent LLMs: GPT-3.5, GPT-4o, Gemini 1.5 Flash, Gemini 1.0 Pro, Claude 3 Opus, Claude 3.5 Sonnet, Llama 3, Gemma 7B, Mistral 7B, and Vicuna-13B using a dataset of 100 carefully designed prompts that probe forced-choice decisions between regions under contextually neutral scenarios. We introduce FAZE, a prompt-based evaluation framework that measures regional bias on a 10-point scale, where higher scores indicate a stronger tendency to favor specific regions. Experimental results reveal substantial variation in bias levels across models, with GPT-3.5 exhibiting the highest bias score (9.5) and Claude 3.5 Sonnet scoring the lowest (2.5). These findings indicate that regional bias can meaningfully undermine the reliability, fairness, and inclusivity of LLM outputs in real-world, cross-cultural applications. This work contributes to AI fairness research by highlighting the importance of inclusive evaluation frameworks and systematic approaches for identifying and mitigating geographic biases in language models.