Detecting AI-Generated Essays in Writing Assessment: Responsible Use and Generalizability Across LLMs

TL;DR

This paper reviews current methods for detecting AI-generated essays, examines their generalizability across different large language models, and offers guidelines for responsible use in writing assessment.

Contribution

It provides empirical analysis of detector generalizability across various LLMs and offers practical guidelines for their responsible deployment.

Findings

Detectors trained on one LLM often do not generalize well to others.

Empirical results highlight the need for retraining detectors for different LLMs.

Guidelines support responsible use of AI detection tools in education.

Abstract

Writing is a foundational literacy skill that underpins effective communication, fosters critical thinking, facilitates learning across disciplines, and enables individuals to organize and articulate complex ideas. Consequently, writing assessment plays a vital role in evaluating language proficiency, communicative effectiveness, and analytical reasoning. The rapid advancement of large language models (LLMs) has made it increasingly easy to generate coherent, high-quality essays, raising significant concerns about the authenticity of student-submitted work. This chapter first provides an overview of the current landscape of detectors for AI-generated and AI-assisted essays, along with guidelines for their responsible use. It then presents empirical analyses to evaluate how well detectors trained on essays from one LLM generalize to identifying essays produced by other LLMs, based on essays generated in response to public GRE writing prompts. These findings provide guidance for developing and retraining detectors for practical applications.