Lessons from Real-World Deployment of a Cognition-Preserving Writing Tool: Students Actively Engage with Critical Thinking and Planning Affordances

TL;DR

This study examines how students in real classroom settings engage with an AI-supported writing tool, VISAR, which provides cognitive scaffolds for argumentative writing, leading to improved learning outcomes and critical AI literacy.

Contribution

It provides empirical evidence on the effective integration of cognition-preserving AI scaffolds in authentic classroom environments, demonstrating positive impacts on writing quality and understanding.

Findings

Students actively engaged with AI scaffolds for planning and idea generation.

Use of scaffolds correlated with better argumentative writing and conceptual understanding.

Students developed emerging critical AI literacy through structured AI interactions.

Abstract

AI-supported writing tools show strong potential for scaffolding students' learning of argumentative writing. Prior work has demonstrated the benefits of AI-supported cognitive scaffolds, such as idea exploration and argument refinement, but how these features function in authentic classroom settings remains underexplored. In this paper, we investigate the classroom integration of an AI-supported writing tool, VISAR. We deployed VISAR in an undergraduate writing course across three sections for one week each over two semesters (49 students total). Using a mixed-methods approach that combines interaction logs, writing artifact analysis, surveys, and interviews, we examine how students used VISAR features in authentic writing tasks. Our findings confirm that students appropriated AI-supported cognitive scaffolds for writing learning and achieved measurable learning gains. While prior studies suggest that students may bypass important cognitive processes when using AI writing assistants, our classroom deployment shows that when systems provide structured supports for planning and targeted generation, students naturally choose to engage with these cognition-preserving scaffolds. These learning-oriented interaction patterns were positively associated with argumentative writing quality, improved conceptual understanding, and emerging critical AI literacy, highlighting the design value of cognition-preserving features in AI writing tools. Together, these findings provide empirical evidence of how AI-supported writing scaffolds operate in authentic classroom contexts and offer design insights for future learning-oriented AI writing tools.