LLM-as-a-tutor in EFL Writing Education: Focusing on Evaluation of Student-LLM Interaction

TL;DR

This paper explores the use of large language models as tutors in EFL writing education, focusing on evaluating student-LLM interactions through pedagogically grounded metrics and expert/learner assessments.

Contribution

It introduces three novel metrics for assessing LLM-based tutoring in EFL writing and integrates pedagogical principles for evaluating feedback quality.

Findings

EFL experts evaluate LLM feedback quality and characteristics.

EFL learners assess their learning outcomes from LLM interaction.

The approach advances tailored LLM tutoring for EFL writing education.

Abstract

In the context of English as a Foreign Language (EFL) writing education, LLM-as-a-tutor can assist students by providing real-time feedback on their essays. However, challenges arise in assessing LLM-as-a-tutor due to differing standards between educational and general use cases. To bridge this gap, we integrate pedagogical principles to assess student-LLM interaction. First, we explore how LLMs can function as English tutors, providing effective essay feedback tailored to students. Second, we propose three metrics to evaluate LLM-as-a-tutor specifically designed for EFL writing education, emphasizing pedagogical aspects. In this process, EFL experts evaluate the feedback from LLM-as-a-tutor regarding quality and characteristics. On the other hand, EFL learners assess their learning outcomes from interaction with LLM-as-a-tutor. This approach lays the groundwork for developing LLMs-as-a-tutor tailored to the needs of EFL learners, advancing the effectiveness of writing education in this context.