How effective are VLMs in assisting humans in inferring the quality of mental models from Multimodal short answers?

TL;DR

This paper introduces MMGrader, a multimodal approach using concept graphs to assess students' mental models from responses, highlighting current models' limitations and potential for educational support.

Contribution

The paper presents MMGrader, a novel method for inferring mental model quality from multimodal student responses using concept graphs, addressing a challenging reasoning task.

Findings

Models achieved about 40% accuracy, below human performance.

Prediction error was approximately 1.1 units.

Model scoring patterns aligned with human scoring.

Abstract

STEM Mental models can play a critical role in assessing students' conceptual understanding of a topic. They not only offer insights into what students know but also into how effectively they can apply, relate to, and integrate concepts across various contexts. Thus, students' responses are critical markers of the quality of their understanding and not entities that should be merely graded. However, inferring these mental models from student answers is challenging as it requires deep reasoning skills. We propose MMGrader, an approach that infers the quality of students' mental models from their multimodal responses using concept graphs as an analytical framework. In our evaluation with 9 openly available models, we found that the best-performing models fall short of human-level performance. This is because they only achieved an accuracy of approximately 40%, a prediction error of 1.1 units, and a scoring distribution fairly aligned with human scoring patterns. With improved accuracy, these can be highly effective assistants to teachers in inferring the mental models of their entire classrooms, enabling them to do so efficiently and help improve their pedagogies more effectively by designing targeted help sessions and lectures that strengthen areas where students collectively demonstrate lower proficiency.