Challenges and Opportunities of Moderating Usage of Large Language Models in Education

TL;DR

This study explores moderated use of large language models in education, proposing methods to encourage critical thinking and reduce unreflected usage, with empirical evidence from student interactions on physics questions.

Contribution

The paper introduces two moderated interaction methods with ChatGPT for educational purposes and empirically evaluates their impact on student engagement and reflection.

Findings

Moderated models led to less unreflected usage than unmoderated ChatGPT.

Neither moderated ChatGPT nor unmoderated ChatGPT matched internet search reflection levels.

Moderation shows potential to support critical thinking in AI-assisted education.

Abstract

The increased presence of large language models (LLMs) in educational settings has ignited debates concerning negative repercussions, including overreliance and inadequate task reflection. Our work advocates moderated usage of such models, designed in a way that supports students and encourages critical thinking. We developed two moderated interaction methods with ChatGPT: hint-based assistance and presenting multiple answer choices. In a study with students (N=40) answering physics questions, we compared the effects of our moderated models against two baseline settings: unmoderated ChatGPT access and internet searches. We analyzed the interaction strategies and found that the moderated versions exhibited less unreflected usage (e.g., copy \& paste) compared to the unmoderated condition. However, neither ChatGPT-supported condition could match the ratio of reflected usage present in internet searches. Our research highlights the potential benefits of moderating language models, showing a research direction toward designing effective AI-supported educational strategies.