Critical Inker: Scaffolding Critical Thinking in AI-Assisted Writing Through Socratic Questioning

TL;DR

Critical Inker is a writing tool that uses Socratic questioning and visual feedback to promote critical thinking and logical analysis in AI-assisted writing, aiming to prevent cognitive deskilling.

Contribution

The paper introduces Critical Inker, a novel system combining Socratic questioning and visual feedback to scaffold critical reflection during AI-assisted writing.

Findings

Achieved 91.2% argument overlap with ground truth annotations.

Attained 87% accuracy in logical validity evaluation.

Conducted a pilot study with early qualitative insights.

Abstract

As Large Language Models (LLMs) increasingly automate writing tasks, there is a growing risk of cognitive deskilling where users offload critical thinking to the system. To address this, we introduce Critical Inker, a writing tool designed to scaffold critical reflection during writing through logical analysis and socratic feedback. We present two methods: (1) A Socratic chatbot using questions to help them realize and fix logical errors in their writing and (2) Visual Feedback, which highlights logical errors in the text without dialog. We detail the technical implementation of the system and evaluate its argument extraction and logical validity accuracy. Our evaluation shows a 91.2% argument overlap with ground truth argument annotations and 87% validity accuracy. Finally, we conducted a small-scale pilot and discuss early qualitative results.