An Empirical Study to Understand How Students Use ChatGPT for Writing Essays

TL;DR

This study investigates how students use ChatGPT for essay writing, revealing usage patterns and their relation to demographics, attitudes, and essay quality to inform educational integration of AI tools.

Contribution

It provides the first empirical analysis of student ChatGPT usage patterns during essay writing and their impact on learning and engagement.

Findings

Gender, race, and self-efficacy predict ChatGPT usage patterns.

Different usage patterns correlate with varying enjoyment and ownership.

Study offers insights for integrating AI tools into writing education.

Abstract

As large language models (LLMs) advance and become widespread, students increasingly turn to systems like ChatGPT for assistance with writing tasks. Educators are concerned with students' usage of ChatGPT beyond cheating; using ChatGPT may reduce their critical engagement with writing, hindering students' learning processes. The negative or positive impact of using LLM-powered tools for writing will depend on how students use them; however, how students use ChatGPT remains largely unknown, resulting in a limited understanding of its impact on learning. To better understand how students use these tools, we conducted an online study $(n=70)$ where students were given an essay-writing task using a custom platform we developed to capture the queries they made to ChatGPT. To characterize their ChatGPT usage, we categorized each of the queries students made to ChatGPT. We then analyzed the relationship between ChatGPT usage and a variety of other metrics, including students' self-perception, attitudes towards AI, and the resulting essay itself. We found that factors such as gender, race, and perceived self-efficacy can help predict different AI usage patterns. Additionally, we found that different usage patterns were associated with varying levels of enjoyment and perceived ownership over the essay. The results of this study contribute to discussions about how writing education should incorporate generative AI-powered tools in the classroom.