GEE! Grammar Error Explanation with Large Language Models

TL;DR

This paper introduces the task of grammar error explanation, evaluates GPT-4's capabilities, and proposes a pipeline that significantly improves explanation accuracy for language learners in German and Chinese.

Contribution

It defines grammar error explanation as a new task, analyzes GPT-4's limitations, and develops a two-step pipeline that enhances explanation quality using fine-tuned models and prompting.

Findings

GPT-4 explains 60.2% of errors with one-shot prompting.

Proposed pipeline achieves over 93% correctness in German and Chinese.

Open-sourcing data and code to foster further research.

Abstract

Grammatical error correction tools are effective at correcting grammatical errors in users' input sentences but do not provide users with \textit{natural language} explanations about their errors. Such explanations are essential for helping users learn the language by gaining a deeper understanding of its grammatical rules (DeKeyser, 2003; Ellis et al., 2006). To address this gap, we propose the task of grammar error explanation, where a system needs to provide one-sentence explanations for each grammatical error in a pair of erroneous and corrected sentences. We analyze the capability of GPT-4 in grammar error explanation, and find that it only produces explanations for 60.2% of the errors using one-shot prompting. To improve upon this performance, we develop a two-step pipeline that leverages fine-tuned and prompted large language models to perform structured atomic token edit extraction, followed by prompting GPT-4 to generate explanations. We evaluate our pipeline on German and Chinese grammar error correction data sampled from language learners with a wide range of proficiency levels. Human evaluation reveals that our pipeline produces 93.9% and 98.0% correct explanations for German and Chinese data, respectively. To encourage further research in this area, we will open-source our data and code.