La Leaderboard: A Large Language Model Leaderboard for Spanish Varieties and Languages of Spain and Latin America

TL;DR

La Leaderboard is an open-source evaluation platform for large language models focusing on Spanish varieties and languages of Spain and Latin America, promoting diversity and reproducibility in LLM development.

Contribution

It introduces the first comprehensive leaderboard for Spanish language varieties, including guidance on evaluation methodology and community-driven development.

Findings

Evaluated 50 models across 66 datasets

Showcased performance differences among language varieties

Provided methodology for sustainable and accessible evaluation

Abstract

Leaderboards showcase the current capabilities and limitations of Large Language Models (LLMs). To motivate the development of LLMs that represent the linguistic and cultural diversity of the Spanish-speaking community, we present La Leaderboard, the first open-source leaderboard to evaluate generative LLMs in languages and language varieties of Spain and Latin America. La Leaderboard is a community-driven project that aims to establish an evaluation standard for everyone interested in developing LLMs for the Spanish-speaking community. This initial version combines 66 datasets in Basque, Catalan, Galician, and different Spanish varieties, showcasing the evaluation results of 50 models. To encourage community-driven development of leaderboards in other languages, we explain our methodology, including guidance on selecting the most suitable evaluation setup for each downstream task. In particular, we provide a rationale for using fewer few-shot examples than typically found in the literature, aiming to reduce environmental impact and facilitate access to reproducible results for a broader research community.