Digital Linguistic Bias in Spanish: Evidence from Lexical Variation in LLMs

TL;DR

This paper investigates how well large language models capture regional lexical differences in Spanish, revealing systematic biases and varying accuracy across dialects using a large-scale, expert-curated lexical database.

Contribution

It provides a comprehensive, large-scale evaluation of dialectal lexical knowledge in LLMs for Spanish, highlighting biases beyond data volume influences.

Findings

Models recognize Spain and Mexico dialects more accurately.

Chilean dialect is particularly challenging for models.

Differences are not solely due to data quantity.

Abstract

This study examines the extent to which Large Language Models (LLMs) capture geographic lexical variation in Spanish, a language that exhibits substantial regional variation. Treating LLMs as virtual informants, we probe their dialectal knowledge using two survey-style question formats: Yes-No questions and multiple-choice questions. To this end, we exploited a large-scale, expert-curated database of Spanish lexical variation. Our evaluation covers more than 900 lexical items across 21 Spanish-speaking countries and is conducted at both the country and dialectal area levels. Across both evaluation formats, the results reveal systematic differences in how LLMs represent Spanish language varieties. Lexical variation associated with Spain, Equatorial Guinea, Mexico & Central America, and the La Plata River is recognized more accurately by the models, while the Chilean variety proves particularly difficult for the models to distinguish. Importantly, differences in the volume of country-level digital resources do not account for these performance patterns, suggesting that factors beyond data quantity shape dialectal representation in LLMs. By providing a fine-grained, large-scale evaluation of geographic lexical variation, this work advances empirical understanding of dialectal knowledge in LLMs and contributes new evidence to discussions of Digital Linguistic Bias in Spanish.