Simulating Validity: Modal Decoupling in MLLM Generated Feedback on Science Drawings

TL;DR

This study investigates the grounding failures of multimodal large language models in providing valid feedback on students' science drawings, revealing prevalent errors and limitations in current approaches.

Contribution

It uncovers modal decoupling issues in off-the-shelf MLLMs and evaluates error types and mitigation strategies using a substantial dataset of student drawings.

Findings

41.3% of feedback instances contained at least one grounding error.

An inventory-list-first workflow reduced some error categories but did not eliminate false absence errors.

Feedback that appears visually grounded offers limited diagnostic value for invalid instances.

Abstract

In science education, students frequently construct hand-drawn visual models of scientific phenomena. These drawings rely on a visual structure where information is encoded through visual objects, their attributes, and relationships. Multimodal large language models (MLLMs) are increasingly used to generate feedback on students' hand-drawn scientific models. However, the validity of such feedback depends on whether model claims are grounded in the specific visual evidence of the student drawing. This study uncovers grounding failures, consistent with modal decoupling, in off-the-shelf MLLM feedback, where outputs remain pedagogically plausible in form while contradicting the drawing or treating depicted elements as missing. Using N = 150 middle school drawings from a kinetic molecular theory unit spanning five modeling tasks and three competence levels, we generated N = 300 feedback instances with GPT-5.1. All outputs were coded for four grounding error types: object mismatch, attribute mismatch, relation mismatch, and false absence. Grounding failures were common: 41.3% of feedback instances contained at least one error. An inventory-list-first workflow reduced several error categories and lowered the overall error rate, but it did not resolve the underlying limitation: approximately one in three outputs remained flawed, with false absence as the dominant failure mode. Moreover, feedback that appears visually grounded offered little diagnostic value for identifying invalid instances. The findings indicate that modal decoupling is a substantial limitation and that valid feedback will require grounding mechanisms beyond common prompting strategies.