Should There be a Teacher In-the-Loop? A Study of Generative AI Personalized Tasks Middle School

TL;DR

This study explores the use of generative AI, specifically ChatGPT, in creating personalized math tasks for middle school students, examining teacher-AI collaboration, efficiency, and student preferences.

Contribution

It provides empirical insights into how teachers interact with generative AI for personalization and the impact on student engagement and task quality.

Findings

Teachers spent significant effort customizing popular culture references.

Students preferred smaller, more specific personalized content.

Teacher-AI collaboration improved problem crafting but was not time-efficient.

Abstract

Adapting instruction to the fine-grained needs of individual students is a powerful application of recent advances in large language models. These generative AI models can create tasks that correspond to students' interests and enact context personalization, enhancing students' interest in learning academic content. However, when there is a teacher in-the-loop creating or modifying tasks with generative AI, it is unclear how efficient this process might be, despite commercial generative AI tools' claims that they will save teachers time. In the present study, we teamed 7 middle school mathematics teachers with ChatGPT to create personalized versions of problems in their curriculum, to correspond to their students' interests. We look at the prompting moves teachers made, their efficiency when creating problems, and the reactions of their 521 7th grade students who received the personalized assignments. We find that having a teacher-in-the-loop results in generative AI-enhanced personalization being enacted at a relatively broad grain size, whereas students tend to prefer a smaller grain size where they receive specific popular culture references that interest them. Teachers spent a lot of effort adjusting popular culture references and addressing issues with the depth or realism of the problems generated, giving higher or lower levels of ownership to the generative AI. Teachers were able to improve in their ability to craft interesting problems in partnership with generative AI, but this process did not appear to become particularly time efficient as teachers learned and reflected on their students' data, iterating their approaches.