Cultural Fidelity in Large-Language Models: An Evaluation of Online Language Resources as a Driver of Model Performance in Value Representation

TL;DR

This study investigates how the availability of digital language resources influences the ability of large-language models to reflect societal values across different languages, revealing significant disparities linked to resource levels.

Contribution

The paper introduces a large, robust dataset of 21 country-language pairs and analyzes the correlation between digital resource availability and LLM performance in value reflection.

Findings

44% of variance in GPT-4o's value reflection linked to digital resources

Error rate over five times higher for low-resource languages

Correlation with GPT-4-turbo increased to 72%

Abstract

The training data for LLMs embeds societal values, increasing their familiarity with the language's culture. Our analysis found that 44% of the variance in the ability of GPT-4o to reflect the societal values of a country, as measured by the World Values Survey, correlates with the availability of digital resources in that language. Notably, the error rate was more than five times higher for the languages of the lowest resource compared to the languages of the highest resource. For GPT-4-turbo, this correlation rose to 72%, suggesting efforts to improve the familiarity with the non-English language beyond the web-scraped data. Our study developed one of the largest and most robust datasets in this topic area with 21 country-language pairs, each of which contain 94 survey questions verified by native speakers. Our results highlight the link between LLM performance and digital data availability in target languages. Weaker performance in low-resource languages, especially prominent in the Global South, may worsen digital divides. We discuss strategies proposed to address this, including developing multilingual LLMs from the ground up and enhancing fine-tuning on diverse linguistic datasets, as seen in African language initiatives.