MAKIEval: A Multilingual Automatic WiKidata-based Framework for Cultural Awareness Evaluation for LLMs
Raoyuan Zhao, Beiduo Chen, Barbara Plank, Michael A. Hedderich

TL;DR
MAKIEval is a multilingual, automatic framework that evaluates cultural awareness in large language models across languages and topics using Wikidata, revealing models' stronger cultural awareness in English.
Contribution
It introduces a scalable, language-agnostic evaluation method for cultural awareness in LLMs leveraging Wikidata, with new metrics and comprehensive multilingual assessment.
Findings
Models show stronger cultural awareness in English.
MAKIEval effectively captures cultural knowledge without manual annotation.
Evaluation spans 13 languages, 19 regions, and 6 topics.
Abstract
Large language models (LLMs) are used globally across many languages, but their English-centric pretraining raises concerns about cross-lingual disparities for cultural awareness, often resulting in biased outputs. However, comprehensive multilingual evaluation remains challenging due to limited benchmarks and questionable translation quality. To better assess these disparities, we introduce MAKIEval, an automatic multilingual framework for evaluating cultural awareness in LLMs across languages, regions, and topics. MAKIEval evaluates open-ended text generation, capturing how models express culturally grounded knowledge in natural language. Leveraging Wikidata's multilingual structure as a cross-lingual anchor, it automatically identifies cultural entities in model outputs and links them to structured knowledge, enabling scalable, language-agnostic evaluation without manual annotation…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
Taxonomy
TopicsCultural Competency in Health Care · Computational and Text Analysis Methods · International Student and Expatriate Challenges
