A Multi-Agent Approach to Validate and Refine LLM-Generated Personalized Math Problems

TL;DR

This paper introduces a multi-agent framework that iteratively refines LLM-generated personalized math problems to improve realism, authenticity, readability, and solvability, based on validation feedback.

Contribution

It formalizes personalization as a generate-validate-revise process with specialized validators, improving problem quality for personalized education.

Findings

Refinement reduces realism and authenticity failures significantly.

Different strategies excel in different validation criteria.

Validator reliability varies, highest on realism, lowest on authenticity.

Abstract

Students benefit from math problems contextualized to their interests. Large language models (LLMs) offer promise for efficient personalization at scale. However, LLM-generated personalized problems may often have problems such as unrealistic quantities and contexts, poor readability, limited authenticity with respect to students' experiences, and occasional mathematical inconsistencies. To alleviate these problems, we propose a multi-agent framework that formalizes personalization as an iterative generate--validate--revise process; we use four specialized validator agents targeting the criteria of solvability, realism, readability, and authenticity, respectively. We evaluate our framework on 600 problems drawn from a popular online mathematics homework platform, ASSISTments, personalizing each problem to a fixed set of 20 student interest topics. We compare three refinement strategies that differ in how validation feedback is coordinated into revisions. Results show that authenticity and realism are the most frequent failure modes in initial LLM-personalized problems, but that a single refinement iteration substantially reduces these failures. We further find that different refinement strategies have different strengths on different criteria. We also assess validator reliability via human evaluation. Results show that reliability is highest on realism and lowest on authenticity, highlighting the need for better evaluation protocols that consider teachers' and students' personal characteristics.