Could an Artificial-Intelligence agent pass an introductory physics course?

TL;DR

This paper investigates whether ChatGPT can pass an introductory physics course by analyzing its responses to assessment questions, revealing it can narrowly pass while displaying common beginner misconceptions.

Contribution

It demonstrates that a state-of-the-art language model can pass an introductory physics exam, highlighting its potential and limitations in understanding physics content.

Findings

ChatGPT narrowly passes the physics course

Exhibits common beginner physics misconceptions

Displays errors similar to novice learners

Abstract

Massive pre-trained language models have garnered attention and controversy due to their ability to generate human-like responses: attention due to their frequent indistinguishability from human-generated phraseology and narratives, and controversy due to the fact that their convincingly presented arguments and facts are frequently simply false. Just how human-like are these responses when it comes to dialogues about physics, in particular about the standard content of introductory physics courses? This study explores that question by having ChatGTP, the pre-eminent language model in 2023, work through representative assessment content of an actual calculus-based physics course and grading the responses in the same way human responses would be graded. As it turns out, ChatGPT would narrowly pass this course while exhibiting many of the preconceptions and errors of a beginning learner.