Do You Know About My Nation? Investigating Multilingual Language Models' Cultural Literacy Through Factual Knowledge

TL;DR

This paper introduces XNationQA, a benchmark for evaluating multilingual language models' cultural literacy across diverse countries and languages, revealing significant disparities and limited cross-lingual knowledge transfer.

Contribution

The paper presents XNationQA, a new multilingual benchmark with novel metrics to assess cultural knowledge, highlighting biases and transfer limitations in current models.

Findings

Models perform better in Western languages but not necessarily for Western countries.

Significant knowledge gaps exist across languages and cultures.

Limited cross-lingual transferability, especially in open-source models.

Abstract

Most multilingual question-answering benchmarks, while covering a diverse pool of languages, do not factor in regional diversity in the information they capture and tend to be Western-centric. This introduces a significant gap in fairly evaluating multilingual models' comprehension of factual information from diverse geographical locations. To address this, we introduce XNationQA for investigating the cultural literacy of multilingual LLMs. XNationQA encompasses a total of 49,280 questions on the geography, culture, and history of nine countries, presented in seven languages. We benchmark eight standard multilingual LLMs on XNationQA and evaluate them using two novel transference metrics. Our analyses uncover a considerable discrepancy in the models' accessibility to culturally specific facts across languages. Notably, we often find that a model demonstrates greater knowledge of cultural information in English than in the dominant language of the respective culture. The models exhibit better performance in Western languages, although this does not necessarily translate to being more literate for Western countries, which is counterintuitive. Furthermore, we observe that models have a very limited ability to transfer knowledge across languages, particularly evident in open-source models.