MateInfoUB: A Real-World Benchmark for Testing LLMs in Competitive, Multilingual, and Multimodal Educational Tasks

TL;DR

This paper introduces MateInfoUB, a bilingual multimodal dataset for testing LLMs in complex CS educational tasks, revealing their strengths and limitations in multilingual and multimodal contexts.

Contribution

It presents a novel bilingual, multimodal dataset based on high-level CS competition questions and systematically evaluates LLMs, highlighting their performance and challenges in educational settings.

Findings

LLMs perform variably across languages and modalities.

Language choice impacts LLM reasoning capabilities.

The dataset enables assessment of LLMs in realistic CS tasks.

Abstract

The rapid advancement of Large Language Models (LLMs) has transformed various domains, particularly computer science (CS) education. These models exhibit remarkable capabilities in code-related tasks and problem-solving, raising questions about their potential and limitations in advanced CS contexts. This study presents a novel bilingual (English-Romanian) multimodal (text and image) dataset of multiple-choice questions derived from a high-level computer science competition. A particularity of our dataset is that the problems are conceived such that some of them are easier solved using reasoning on paper, while for others writing code is more efficient. We systematically evaluate State of The Art LLMs on this dataset, analyzing their performance on theoretical programming tasks. Our findings reveal the strengths and limitations of current LLMs, including the influence of language choice (English vs. Romanian), providing insights into their applicability in CS education and competition settings. We also address critical ethical considerations surrounding educational integrity and the fairness of assessments in the context of LLM usage. These discussions aim to inform future educational practices and policies. To support further research, our dataset will be made publicly available in both English and Romanian. Additionally, we release an educational application tailored for Romanian students, enabling them to self-assess using the dataset in an interactive and practice-oriented environment.