Evaluating Large Language Models for automatic analysis of teacher simulations

TL;DR

This paper evaluates the effectiveness of Large Language Models, specifically DeBERTaV3 and Llama 3, in automatically analyzing responses in digital simulations for teacher training, highlighting their varying performance in identifying user behaviors.

Contribution

The study compares LLMs for analyzing teacher simulation responses, revealing Llama 3's superior stability in detecting new characteristics over DeBERTaV3.

Findings

Llama 3 outperforms DeBERTaV3 in identifying new response characteristics.

Performance of LLMs varies significantly depending on the characteristic to identify.

Llama 3 shows more stable performance in dynamic educational scenarios.

Abstract

Digital Simulations (DS) provide safe environments where users interact with an agent through conversational prompts, providing engaging learning experiences that can be used to train teacher candidates in realistic classroom scenarios. These simulations usually include open-ended questions, allowing teacher candidates to express their thoughts but complicating an automatic response analysis. To address this issue, we have evaluated Large Language Models (LLMs) to identify characteristics (user behaviors) in the responses of DS for teacher education. We evaluated the performance of DeBERTaV3 and Llama 3, combined with zero-shot, few-shot, and fine-tuning. Our experiments discovered a significant variation in the LLMs' performance depending on the characteristic to identify. Additionally, we noted that DeBERTaV3 significantly reduced its performance when it had to identify new characteristics. In contrast, Llama 3 performed better than DeBERTaV3 in detecting new characteristics and showing more stable performance. Therefore, in DS where teacher educators need to introduce new characteristics because they change depending on the simulation or the educational objectives, it is more recommended to use Llama 3. These results can guide other researchers in introducing LLMs to provide the highly demanded automatic evaluations in DS.