Evaluating Large Language Models with Tests of Spanish as a Foreign Language: Pass or Fail?

TL;DR

This paper assesses the ability of large language models to understand Spanish through a benchmark resembling Spanish exams, revealing they perform well but still lack native-level grammatical competence.

Contribution

It introduces TELEIA, a new benchmark for evaluating LLMs on Spanish language tasks, highlighting their strengths and limitations in understanding non-English languages.

Findings

LLMs perform well on Spanish comprehension tasks.

They are still below native speakers in grammatical accuracy.

The TELEIA benchmark provides a comprehensive Spanish language evaluation.

Abstract

Large Language Models (LLMs) have been profusely evaluated on their ability to answer questions on many topics and their performance on different natural language understanding tasks. Those tests are usually conducted in English, but most LLM users are not native English speakers. Therefore, it is of interest to analyze how LLMs understand other languages at different levels: from paragraphs to morphems. In this paper, we evaluate the performance of state-of-the-art LLMs in TELEIA, a recently released benchmark with similar questions to those of Spanish exams for foreign students, covering topics such as reading comprehension, word formation, meaning and compositional semantics, and grammar. The results show that LLMs perform well at understanding Spanish but are still far from achieving the level of a native speaker in terms of grammatical competence.