"I Wrote, I Paused, I Rewrote" Teaching LLMs to Read Between the Lines of Student Writing

TL;DR

This study investigates enhancing large language models' feedback on student writing by incorporating process data from keystrokes and revisions, leading to more personalized and meaningful guidance.

Contribution

The paper introduces a digital writing tool capturing writing process data and demonstrates that process-aware LLM feedback improves student perception and aligns with quality improvements.

Findings

Students preferred process-aware feedback over final-only feedback.

Certain edits like adding content or reorganizing correlated with higher quality scores.

Process-aware feedback was perceived as more relatable and supportive.

Abstract

Large language models(LLMs) like Gemini are becoming common tools for supporting student writing. But most of their feedback is based only on the final essay missing important context about how that text was written. In this paper, we explore whether using writing process data, collected through keystroke logging and periodic snapshots, can help LLMs give feedback that better reflects how learners think and revise while writing. We built a digital writing tool that captures both what students type and how their essays evolve over time. Twenty students used this tool to write timed essays, which were then evaluated in two ways: (i) LLM generated feedback using both the final essay and the full writing trace, and (ii) After the task, students completed surveys about how useful and relatable they found the feedback. Early results show that learners preferred the process-aware LLM feedback, finding it more in tune with their own thinking. We also found that certain types of edits, like adding new content or reorganizing paragraphs, aligned closely with higher scores in areas like coherence and elaboration. Our findings suggest that making LLMs more aware of the writing process can lead to feedback that feels more meaningful, personal, and supportive.