Challenges for AI in Multimodal STEM Assessments: a Human-AI Comparison

TL;DR

This study compares human and AI performance on multimodal STEM questions, revealing AI struggles with visual components and offering insights for designing assessments that maintain academic integrity.

Contribution

Introduces a new dataset of STEM questions and analyzes how multimodal features affect AI versus human performance, providing guidance for assessment design.

Findings

AI achieves 58.5% accuracy with best prompting strategies

Humans outperform AI on visual questions consistently

AI performance varies with subject and question features

Abstract

Generative AI systems have rapidly advanced, with multimodal input capabilities enabling reasoning beyond text-based tasks. In education, these advancements could influence assessment design and question answering, presenting both opportunities and challenges. To investigate these effects, we introduce a high-quality dataset of 201 university-level STEM questions, manually annotated with features such as image type, role, problem complexity, and question format. Our study analyzes how these features affect generative AI performance compared to students. We evaluate four model families with five prompting strategies, comparing results to the average of 546 student responses per question. Although the best model correctly answers on average 58.5 % of the questions using majority vote aggregation, human participants consistently outperform AI on questions involving visual components. Interestingly, human performance remains stable across question features but varies by subject, whereas AI performance is susceptible to both subject matter and question features. Finally, we provide actionable insights for educators, demonstrating how question design can enhance academic integrity by leveraging features that challenge current AI systems without increasing the cognitive burden for students.