ELMES: An Automated Framework for Evaluating Large Language Models in Educational Scenarios

TL;DR

ELMES is an open-source framework that automates the evaluation of large language models in educational scenarios, addressing the lack of specialized assessment metrics and enabling flexible, pedagogically relevant benchmarking.

Contribution

We introduce ELMES, a modular, hybrid evaluation framework tailored for assessing LLMs in education, with a novel LLM-as-a-Judge methodology and scenario-specific metrics.

Findings

Distinct capability profiles for different models across scenarios

ELMES effectively quantifies pedagogical metrics objectively

Framework reduces barriers for educational LLM evaluation

Abstract

The emergence of Large Language Models (LLMs) presents transformative opportunities for education, generating numerous novel application scenarios. However, significant challenges remain: evaluation metrics vary substantially across different educational scenarios, while many emerging scenarios lack appropriate assessment metrics. Current benchmarks predominantly measure general intelligence rather than pedagogical capabilities. To address this gap, we introduce ELMES, an open-source automated evaluation framework specifically designed for assessing LLMs in educational settings. ELMES features a modular architecture that enables researchers to create dynamic, multi-agent dialogues through simple configuration files, facilitating flexible scenario design without requiring extensive programming expertise. The framework incorporates a hybrid evaluation engine that objectively quantifies traditionally subjective pedagogical metrics using an LLM-as-a-Judge methodology. We conduct systematic benchmarking of state-of-the-art LLMs across four critical educational scenarios: Knowledge Point Explanation, Guided Problem-Solving Teaching, Interdisciplinary Lesson Plan Generation, and Contextualized Question Generation, employing fine-grained metrics developed in collaboration with education specialists. Our results demonstrate distinct capability distributions among models, revealing context-specific strengths and limitations. ELMES provides educators and researchers with an accessible evaluation framework that significantly reduces adaptation barriers for diverse educational applications while advancing the practical implementation of LLMs in pedagogy. The framework is publicly available at \emph{https://github.com/sii-research/elmes.git}.