Grammatical Error Feedback: An Implicit Evaluation Approach

TL;DR

This paper introduces an implicit evaluation method for grammatical error feedback (GEF) using large language models, eliminating manual annotations and assessing feedback quality through a novel lineup approach.

Contribution

It presents a new implicit evaluation framework for GEF leveraging LLMs and a lineup method to assess feedback without manual annotations.

Findings

The lineup approach effectively evaluates GEF quality.

LLMs can match feedback and essay representations accurately.

The study highlights the importance of foil selection in feedback evaluation.

Abstract

Grammatical feedback is crucial for consolidating second language (L2) learning. Most research in computer-assisted language learning has focused on feedback through grammatical error correction (GEC) systems, rather than examining more holistic feedback that may be more useful for learners. This holistic feedback will be referred to as grammatical error feedback (GEF). In this paper, we present a novel implicit evaluation approach to GEF that eliminates the need for manual feedback annotations. Our method adopts a grammatical lineup approach where the task is to pair feedback and essay representations from a set of possible alternatives. This matching process can be performed by appropriately prompting a large language model (LLM). An important aspect of this process, explored here, is the form of the lineup, i.e., the selection of foils. This paper exploits this framework to examine the quality and need for GEC to generate feedback, as well as the system used to generate feedback, using essays from the Cambridge Learner Corpus.