MCEval: A Dynamic Framework for Fair Multilingual Cultural Evaluation of LLMs

TL;DR

MCEval is a comprehensive multilingual framework that evaluates large language models' cultural awareness and bias across 13 cultures and languages, revealing disparities linked to language-culture alignment.

Contribution

It introduces the first dynamic, multilingual cultural evaluation framework with causal analysis capabilities for assessing LLMs' cultural understanding.

Findings

Performance varies across linguistic scenarios.

Optimal cultural performance depends on language-culture alignment.

English-centric approaches can cause fairness issues.

Abstract

Large language models exhibit cultural biases and limited cross-cultural understanding capabilities, particularly when serving diverse global user populations. We propose MCEval, a novel multilingual evaluation framework that employs dynamic cultural question construction and enables causal analysis through Counterfactual Rephrasing and Confounder Rephrasing. Our comprehensive evaluation spans 13 cultures and 13 languages, systematically assessing both cultural awareness and cultural bias across different linguistic scenarios. The framework provides 39,897 cultural awareness instances and 17,940 cultural bias instances. Experimental results reveal performance disparities across different linguistic scenarios, demonstrating that optimal cultural performance is not only linked to training data distribution, but also is related to language-culture alignment. The evaluation results also expose the fairness issue, where approaches appearing successful in the English scenario create substantial disadvantages. MCEval represents the first comprehensive multilingual cultural evaluation framework that provides deeper insights into LLMs' cultural understanding.