FEANEL: A Benchmark for Fine-Grained Error Analysis in K-12 English Writing

TL;DR

This paper introduces FEANEL, a benchmark with expert-annotated student essays and error taxonomy to evaluate and improve LLMs' ability to provide detailed feedback for K-12 English writing.

Contribution

It presents the FEANEL benchmark, including a large annotated dataset and taxonomy, to assess and enhance LLMs' error analysis and pedagogical skills in education.

Findings

Current LLMs show significant gaps in fine-grained error analysis.

The benchmark reveals specific areas where LLMs need improvement.

Experimental results highlight the need for advanced methods for educational feedback.

Abstract

Large Language Models (LLMs) have transformed artificial intelligence, offering profound opportunities for educational applications. However, their ability to provide fine-grained educational feedback for K-12 English writing remains underexplored. In this paper, we challenge the error analysis and pedagogical skills of LLMs by introducing the problem of Fine-grained Error Analysis for English Learners and present the Fine-grained Error ANalysis for English Learners (FEANEL) Benchmark. The benchmark comprises 1,000 essays written by elementary and secondary school students, and a well-developed English writing error taxonomy. Each error is annotated by language education experts and categorized by type, severity, and explanatory feedback, using a part-of-speech-based taxonomy they co-developed. We evaluate state-of-the-art LLMs on the FEANEL Benchmark to explore their error analysis and pedagogical abilities. Experimental results reveal significant gaps in current LLMs' ability to perform fine-grained error analysis, highlighting the need for advancements in particular methods for educational applications.