Multimodal large language models and physics visual tasks: comparative analysis of performance and costs

TL;DR

This study benchmarks 15 multimodal large language models on physics visual tasks, revealing significant variability in performance and costs, and providing insights for educational stakeholders to make informed model choices.

Contribution

It offers a comparative analysis of multiple MLLMs on physics conceptual assessments, highlighting performance variability and cost considerations for educational applications.

Findings

Performance ranges from 21% to 81.5%.

Cheaper models can be sufficient for certain tasks.

Expensive models do not always outperform cheaper ones.

Abstract

Multimodal large language models (MLLMs) capable of processing both text and visual inputs are increasingly being explored for uses in physics education, such as tutoring, formative assessment, and grading. This study evaluates a range of publicly available MLLMs on a set of standardized, image-based physics research-based conceptual assessments (concept inventories). We benchmark 15 models from three major providers (Anthropic, Google, and OpenAI) across 102 physics items, focusing on two main questions: (1) How well do these models perform on conceptual physics tasks involving visual representations? and (2) What are the financial costs associated with their use? The results show high variability in both performance and cost. The performance of the tested models ranges from 81.5% to as low as 21%. We also found that expensive models do not always outperform cheaper ones and that, depending on the demands of the context, cheaper models may be sufficiently capable for some tasks. This is especially relevant in contexts where financial resources are limited or for large-scale educational implementation of MLLMs. By providing these analyses, our aim is to inform teachers, institutions, and other educational stakeholders so that they can make evidence-based decisions about the selection of models for use in AI-supported physics education.