Mathematics Teachers Interactions with a Multi-Agent System for Personalized Problem Generation

TL;DR

This study explores a multi-agent system that assists middle school math teachers in creating personalized problems using large language models and specialized evaluative agents.

Contribution

It introduces a teacher-in-the-loop multi-agent framework for generating and evaluating personalized math problems, highlighting real-world context customization.

Findings

Teachers and students desired more real-world context modifications.

Few issues with realism and readability were present in final problems.

The system supports teacher control in personalized problem generation.

Abstract

Large language models can increasingly adapt educational tasks to learners characteristics. In the present study, we examine a multi-agent teacher-in-the-loop system for personalizing middle school math problems. The teacher enters a base problem and desired topic, the LLM generates the problem, and then four AI agents evaluate the problem using criteria that each specializes in (mathematical accuracy, authenticity, readability, and realism). Eight middle school mathematics teachers created 212 problems in ASSISTments using the system and assigned these problems to their students. We find that both teachers and students wanted to modify the fine-grained personalized elements of the real-world context of the problems, signaling issues with authenticity and fit. Although the agents detected many issues with realism as the problems were being written, there were few realism issues noted by teachers and students in the final versions. Issues with readability and mathematical hallucinations were also somewhat rare. Implications for multi-agent systems for personalization that support teacher control are given.