The Crutch or the Ceiling? How Different Generations of LLMs Shape EFL Student Writings

TL;DR

This study investigates how different generations of LLMs influence EFL students' writing development, revealing that advanced models can improve scores but may mask true proficiency, emphasizing the need for pedagogical shifts.

Contribution

It provides a comparative analysis of LLM assistance before and after ChatGPT, highlighting impacts on assessment scores, lexical diversity, and the importance of pedagogical adaptation.

Findings

Advanced LLMs increase assessment scores for lower-proficiency learners.

Higher LLM assistance correlates negatively with human expert ratings.

Surface fluency improves, but deep coherence and true learning may be compromised.

Abstract

The rapid evolution of Large Language Models (LLMs) has made them powerful tools for enhancing student writing. This study explores the extent and limitations of LLMs in assisting secondary-level English as a Foreign Language (EFL) students with their writing tasks. While existing studies focus on output quality, our research examines the developmental shift in LLMs and their impact on EFL students, assessing whether smarter models act as true scaffolds or mere compensatory crutches. To achieve this, we analyse student compositions assisted by LLMs before and after ChatGPT's release, using both expert qualitative scoring and quantitative metrics (readability tests, Pearson's correlation coefficient, MTLD, and others). Our results indicate that advanced LLMs boost assessment scores and lexical diversity for lower-proficiency learners, potentially masking their true ability. Crucially, increased LLM assistance correlated negatively with human expert ratings, suggesting surface fluency without deep coherence. To transform AI-assisted practice into genuine learning, pedagogy must shift from focusing on output quality to verifying the learning process. Educators should align AI functions, specifically differentiating ideational scaffolding from textual production, within the learner's Zone of Proximal Development.