The Death of the Short-Form Physics Essay in the Coming AI Revolution

TL;DR

AI language models can produce high-quality, full short-form physics essays that can achieve top grades, posing a significant threat to traditional assessment methods in physics education.

Contribution

This work provides empirical evidence that current AI models can generate physics essays that pass university assessments at the highest level, challenging the integrity of essay-based evaluation.

Findings

AI essays scored around 71%, matching university averages.

AI-generated essays are indistinguishable from human work in grading.

Plagiarism scores are low, indicating originality in AI outputs.

Abstract

The latest AI language modules can produce original, high quality full short-form ($300$-word) Physics essays within seconds. These technologies such as ChatGPT and davinci-003 are freely available to anyone with an internet connection. In this work, we present evidence of AI generated short-form essays achieving first-class grades on an essay writing assessment from an accredited, current university Physics module. The assessment requires students answer five open-ended questions with a short, $300$-word essay each. Fifty AI answers were generated to create ten submissions that were independently marked by five separate markers. The AI generated submissions achieved an average mark of $71 \pm 2 \%$, in strong agreement with the current module average of $71 \pm 5 %$. A typical AI submission would therefore most-likely be awarded a First Class, the highest classification available at UK universities. Plagiarism detection software returned a plagiarism score between $2 \pm 1$% (Grammarly) and $7 \pm 2$% (TurnitIn). We argue that these results indicate that current AI MLPs represent a significant threat to the fidelity of short-form essays as an assessment method in Physics courses.