Can ChatGPT Pass a Theory of Computing Course?

TL;DR

This study investigates whether ChatGPT can pass a Theory of Computing course by evaluating its performance on exams and sample questions, revealing it can pass but struggles with complex open-ended responses.

Contribution

The paper demonstrates that ChatGPT can pass a ToC course's exams and provides a detailed analysis of its strengths and limitations in understanding formal concepts.

Findings

ChatGPT can pass ToC course exams.

It handles simple questions well but makes nonsensical claims in proofs.

Performance varies with question type.

Abstract

Large Language Models (LLMs) have had considerable difficulty when prompted with mathematical questions, especially those within theory of computing (ToC) courses. In this paper, we detail two experiments regarding our own ToC course and the ChatGPT LLM. For the first, we evaluated ChatGPT's ability to pass our own ToC course's exams. For the second, we created a database of sample ToC questions and responses to accommodate other ToC offerings' choices for topics and structure. We scored each of ChatGPT's outputs on these questions. Overall, we determined that ChatGPT can pass our ToC course, and is adequate at understanding common formal definitions and answering "simple"-style questions, e.g., true/false and multiple choice. However, ChatGPT often makes nonsensical claims in open-ended responses, such as proofs.