MATHWELL: Generating Educational Math Word Problems Using Teacher Annotations

TL;DR

MATHWELL is a fine-tuned 70B language model designed to generate educational math word problems that are solvable, accurate, and appropriate for K-8 students, outperforming existing models in quality and safety.

Contribution

The paper introduces MATHWELL, the first K-8 targeted math word problem generator fine-tuned with teacher annotations for educational appropriateness.

Findings

MATHWELL produces more solvable and accurate problems than public models.

MATHWELL matches GPT-4 in problem quality but with better appropriateness for K-8.

MATHWELL avoids generating harmful questions and matches desired reading levels.

Abstract

Math word problems are critical K-8 educational tools, but writing them is time consuming and requires extensive expertise. To be educational, problems must be solvable, have accurate answers, and, most importantly, be educationally appropriate. We propose that language models have potential to support K-8 math education by automatically generating word problems. However, evaluating educational appropriateness is hard to quantify. We fill this gap by having teachers evaluate problems generated by LLMs, who find existing models and data often fail to be educationally appropriate. We then explore automatically generating educational word problems, ultimately using our expert annotations to finetune a 70B language model. Our model, MATHWELL, is the first K-8 word problem generator targeted at educational appropriateness. Further expert studies find MATHWELL generates problems far more solvable, accurate, and appropriate than public models. MATHWELL also matches GPT-4's problem quality while attaining more appropriate reading levels for K-8 students and avoiding generating harmful questions.