The Veln(ia)s is in the Details: Evaluating LLM Judgment on Latvian and Lithuanian Short Answer Matching

TL;DR

This study evaluates the ability of various large language models to accurately perform short answer matching in Latvian and Lithuanian, revealing that larger models excel while smaller models vary in performance, especially with subtle text differences.

Contribution

Introduces new datasets for Latvian and Lithuanian short answer matching and benchmarks multiple LLMs, highlighting their strengths and limitations in subtle text difference detection.

Findings

Larger LLMs like QWEN2.5 72b perform near-perfectly in answer matching.

Smaller models show variable performance, improved with few-shot learning.

Mistral Nemo 12b underperforms in detecting subtle text alterations.

Abstract

In this work, we address the challenge of evaluating large language models (LLMs) on the short answer matching task for Latvian and Lithuanian languages. We introduce novel datasets consisting of 502 Latvian and 690 Lithuanian question-answer pairs. For each question-answer pair, we generated matched and non-matched answers using a set of alteration rules specifically designed to introduce small but meaningful changes in the text. These generated answers serve as test cases to assess the ability of LLMs to detect subtle differences in matching of the original answers. A subset of the datasets was manually verified for quality and accuracy. Our results show that while larger LLMs, such as QWEN2.5 72b and LLaMa3.1 70b, demonstrate near-perfect performance in distinguishing matched and non-matched answers, smaller models show more variance. For instance, LLaMa3.1 8b and EuroLLM 9b benefited from few-shot examples, while Mistral Nemo 12b underperformed on detection of subtle text alteration, particularly in Lithuanian, even with additional examples. QWEN2.5 7b and Mistral 7b were able to obtain a strong and comparable performance to the larger 70b models in zero and few shot experiments. Moreover, the performance of Mistral 7b was weaker in few shot experiments.