"Can You See Me Think?" Grounding LLM Feedback in Keystrokes and Revision Patterns

TL;DR

This paper investigates whether large language models can provide more insightful feedback on student writing by analyzing writing process data like keystrokes and revision patterns, beyond just the final essay.

Contribution

It introduces a method to incorporate writing process data into LLM feedback, demonstrating improved structural evaluation and process-sensitive justification over traditional final-product-only approaches.

Findings

Enhanced structural evaluation in feedback

Greater process-sensitive justification

Minimal change in rubric scores

Abstract

As large language models (LLMs) increasingly assist in evaluating student writing, researchers have begun questioning whether these models can be cognitively grounded, that is, whether they can attend not just to the final product, but to the process by which it was written. In this study, we explore how incorporating writing process data, specifically keylogs and time-stamped snapshots, affects the quality of LLM-generated feedback. We conduct an ablation study on 52 student essays comparing feedback generated with access to only the final essay (C1) and feedback that also incorporates keylogs and time-stamped snapshots (C2). While rubric scores changed minimally, C2 feedback demonstrated significantly improved structural evaluation and greater process-sensitive justification.