Moving Beyond Review: Applying Language Models to Planning and Translation in Reflection

TL;DR

This paper introduces Pensée, a novel LLM-based tool that supports students during planning and translation stages of reflective writing, leading to deeper reflection and better structure in their work.

Contribution

It is the first to apply LLMs to support the planning and translation phases of reflective writing based on the Cognitive Process Theory, demonstrating improved reflection quality.

Findings

Support during planning and translation increases reflection depth.

Learners show improved structural quality with AI support.

Effects on reflection diminish in delayed post-test.

Abstract

Reflective writing is known to support the development of students' metacognitive skills, yet learners often struggle to engage in deep reflection, limiting learning gains. Although large language models (LLMs) have been shown to improve writing skills, their use as conversational agents for reflective writing has produced mixed results and has largely focused on providing feedback on reflective texts, rather than support during planning and organizing. In this paper, inspired by the Cognitive Process Theory of writing (CPT), we propose the first application of LLMs to the planning and translation steps of reflective writing. We introduce Pens\'ee, a tool to explore the effects of explicit AI support during these stages by scaffolding structured reflection planning using a conversational agent, and supporting translation by automatically extracting key concepts. We evaluate Pens\'ee in a controlled between-subjects experiment (N=93), manipulating AI support across writing phases. Results show significantly greater reflection depth and structural quality when learners receive support during planning and translation stages of CPT, though these effects reduce in a delayed post-test. Analyses of learner behavior and perceptions further illustrate how CPT-aligned conversational support shapes reflection processes and learner experience, contributing empirical evidence for theory-driven uses of LLMs in AI-supported reflective writing.