A Dialogue-Based Framework for Correcting Multimodal Errors in AI-Assisted STEM Education

TL;DR

This paper evaluates LLMs on multimodal physics problems, identifies specific failure modes, and demonstrates a dialogue-based intervention that significantly improves error correction without retraining models.

Contribution

It introduces a structured multimodal dialogue framework that effectively corrects errors in LLMs handling multimodal STEM content, addressing a key limitation in AI tutoring.

Findings

LLMs perform well on text-only physics problems (96% accuracy).

Performance drops on multimodal problems due to specific failure modes.

The dialogue intervention corrected 82% of errors, especially visual processing errors.

Abstract

Large Language Models (LLMs) are democratizing access to personalized tutoring; however, their effectiveness is hindered by challenges in processing multimodal content, which limits AI's potential to provide equitable, high-quality STEM support. This study evaluates LLM performance on multimodal physics problems, identifies specific failure modes through an empirical error taxonomy, and tests practical interventions designed to overcome multimodal processing limitations. We assessed three publicly available LLMs (Claude, Gemini, and ChatGPT) on multimodal physics problems from the OpenStax database and compared the results with text-only performance. An empirically derived error taxonomy was developed through pilot testing, followed by evaluation of a structured multimodal dialogue intervention. All three models achieved near-ceiling accuracy (96%) on text-only physics problems. Performance declined substantially on multimodal problems, consistent with what we term the Multimodal Interference Effect. Error analysis identified four failure modes: visual processing errors, context misinterpretation, mathematical computational errors, and hybrid errors, with visual processing errors being the most prevalent. The structured dialogue intervention corrected 82% of errors overall; visual processing errors were corrected at 100% across all models. Educators and students can implement these interventions immediately, requiring no model retraining, to improve AI tutoring reliability on image-rich STEM content, advancing equitable access to high-quality learning support.