The Aftermath of DrawEduMath: Vision Language Models Underperform with Struggling Students and Misdiagnose Errors

TL;DR

This study evaluates 11 vision-language models on a math education benchmark, revealing significant underperformance in understanding and diagnosing student errors, especially for struggling students, highlighting the need for specialized development.

Contribution

It provides the first extensive, year-long analysis of VLMs in educational contexts, emphasizing their limitations in error diagnosis and student support.

Findings

All models underperform on student work requiring pedagogical help.

Models struggle most with questions about student error assessment.

VLMs need alternative development incentives for educational applications.

Abstract

Effective mathematics education requires identifying and responding to students' mistakes. For AI to support pedagogical applications, models must perform well across different levels of student proficiency. Our work provides an extensive, year-long snapshot of how 11 vision-language models (VLMs) perform on DrawEduMath, a QA benchmark involving real students' handwritten, hand-drawn responses to math problems. We find that models' weaknesses concentrate on a core component of math education: student error. All evaluated VLMs underperform when describing work from students who require more pedagogical help, and across all QA, they struggle the most on questions related to assessing student error. Thus, while VLMs may be optimized to be math problem solving experts, our results suggest that they require alternative development incentives to adequately support educational use cases.