More Robots are Coming: Large Multimodal Models (ChatGPT) can Solve Visually Diverse Images of Parsons Problems

TL;DR

Large multimodal models like GPT-4V can effectively solve visually diverse Parsons problems, raising questions about their impact on academic integrity and assessment strategies in computing education.

Contribution

This study evaluates the performance of GPT-4V and Bard on visual Parsons problems, highlighting GPT-4V's high success rate and implications for education.

Findings

GPT-4V solved 96.7% of visual Parsons problems.

Bard solved 69.2% of problems and faced hallucination issues.

Visual problems alone may not prevent academic integrity violations.

Abstract

The advent of large language models is reshaping computing education. Recent research has demonstrated that these models can produce better explanations than students, answer multiple-choice questions at or above the class average, and generate code that can pass automated tests in introductory courses. These capabilities have prompted instructors to rapidly adapt their courses and assessment methods to accommodate changes in learning objectives and the potential for academic integrity violations. While some scholars have advocated for the integration of visual problems as a safeguard against the capabilities of language models, new multimodal language models now have vision and language capabilities that may allow them to analyze and solve visual problems. In this paper, we evaluate the performance of two large multimodal models on visual assignments, with a specific focus on Parsons problems presented across diverse visual representations. Our results show that GPT-4V solved 96.7\% of these visual problems, struggling minimally with a single Parsons problem. Conversely, Bard performed poorly by only solving 69.2\% of problems, struggling with common issues like hallucinations and refusals. These findings suggest that merely transitioning to visual programming problems might not be a panacea to issues of academic integrity in the generative AI era.