Performance of ChatGPT on the Test of Understanding Graphs in Kinematics

TL;DR

This study evaluates ChatGPT-4's ability to interpret kinematic graphs, revealing it performs comparably to high-school students but struggles with visual graph comprehension, highlighting cautious use in educational settings.

Contribution

First assessment of ChatGPT-4's performance on graph understanding in physics, emphasizing its strengths in reasoning and limitations in visual perception.

Findings

ChatGPT performs similarly to high-school students in graph interpretation.

It demonstrates strong reasoning but has difficulty visually 'seeing' graphs.

Caution is advised when using ChatGPT for educational or assistive purposes involving graphs.

Abstract

The well-known artificial intelligence-based chatbot ChatGPT-4 has become able to process image data as input in October 2023. We investigated its performance on the Test of Understanding Graphs in Kinematics to inform the physics education community of the current potential of using ChatGPT in the education process, particularly on tasks that involve graphical interpretation. We found that ChatGPT, on average, performed similarly to students taking a high-school level physics course, but with important differences in the distribution of the correctness of its responses, as well as in terms of the displayed "reasoning" and "visual" abilities. While ChatGPT was very successful at proposing productive strategies for solving the tasks on the test and expressed correct "reasoning" in most of its responses, it had difficulties correctly "seeing" graphs. We suggest that, based on its performance, caution and a critical approach are needed if one intends to use it in the role of a tutor, a model of a student, or a tool for assisting vision-impaired persons in the context of kinematics graphs.