DLAMA: A Framework for Curating Culturally Diverse Facts for Probing the Knowledge of Pretrained Language Models

TL;DR

This paper introduces DLAMA-v1, a culturally diverse benchmark for probing multilingual and monolingual language models' factual knowledge, revealing biases towards Western facts and cultural relevance in predictions.

Contribution

It presents a novel framework for curating culturally diverse factual triples and constructs DLAMA-v1, addressing biases in existing benchmarks and enabling fairer evaluation of language models.

Findings

mBERT performs better on Western facts than non-Western ones.

Monolingual models excel on culturally proximate facts.

Models tend to predict culturally or geographically relevant labels.

Abstract

A few benchmarking datasets have been released to evaluate the factual knowledge of pretrained language models. These benchmarks (e.g., LAMA, and ParaRel) are mainly developed in English and later are translated to form new multilingual versions (e.g., mLAMA, and mParaRel). Results on these multilingual benchmarks suggest that using English prompts to recall the facts from multilingual models usually yields significantly better and more consistent performance than using non-English prompts. Our analysis shows that mLAMA is biased toward facts from Western countries, which might affect the fairness of probing models. We propose a new framework for curating factual triples from Wikidata that are culturally diverse. A new benchmark DLAMA-v1 is built of factual triples from three pairs of contrasting cultures having a total of 78,259 triples from 20 relation predicates. The three pairs comprise facts representing the (Arab and Western), (Asian and Western), and (South American and Western) countries respectively. Having a more balanced benchmark (DLAMA-v1) supports that mBERT performs better on Western facts than non-Western ones, while monolingual Arabic, English, and Korean models tend to perform better on their culturally proximate facts. Moreover, both monolingual and multilingual models tend to make a prediction that is culturally or geographically relevant to the correct label, even if the prediction is wrong.