RETUYT-INCO at BEA 2025 Shared Task: How Far Can Lightweight Models Go in AI-powered Tutor Evaluation?

TL;DR

This paper demonstrates that small, lightweight models under 1 billion parameters can effectively participate in AI-powered tutor evaluation tasks, maintaining competitive performance with limited computational resources.

Contribution

It shows that compact models can achieve competitive results in tutor evaluation shared tasks, highlighting their suitability for resource-constrained environments.

Findings

Lightweight models achieved near-winning performance in shared tasks.

Models under 1B parameters can run on low-budget GPUs or without GPUs.

Performance gap between small models and top models ranged from 6.46 to 13.13 F1 points.

Abstract

In this paper, we present the RETUYT-INCO participation at the BEA 2025 shared task. Our participation was characterized by the decision of using relatively small models, with fewer than 1B parameters. This self-imposed restriction tries to represent the conditions in which many research labs or institutions are in the Global South, where computational power is not easily accessible due to its prohibitive cost. Even under this restrictive self-imposed setting, our models managed to stay competitive with the rest of teams that participated in the shared task. According to the $exact\ F_1$ scores published by the organizers, the performance gaps between our models and the winners were as follows: $6.46$ in Track 1; $10.24$ in Track 2; $7.85$ in Track 3; $9.56$ in Track 4; and $13.13$ in Track 5. Considering that the minimum difference with a winner team is $6.46$ points -- and the maximum difference is $13.13$ -- according to the $exact\ F_1$ score, we find that models with a size smaller than 1B parameters are competitive for these tasks, all of which can be run on computers with a low-budget GPU or even without a GPU.