ChatGPT Participates in a Computer Science Exam

TL;DR

This study evaluates ChatGPT's performance on a university computer science exam, finding it narrowly passes and highlighting its capabilities and limitations in understanding complex academic tasks.

Contribution

First systematic assessment of ChatGPT's exam performance in computer science, comparing GPT-3.5 and GPT-4, and analyzing its potential and limitations.

Findings

ChatGPT narrowly passes the exam with 20.5/40 points.

GPT-4 outperforms GPT-3.5 by 17%, reaching average student performance.

Performance is based on exam questions similar to training data, not true understanding.

Abstract

We asked ChatGPT to participate in an undergraduate computer science exam on ''Algorithms and Data Structures''. The program was evaluated on the entire exam as posed to the students. We hand-copied its answers onto an exam sheet, which was subsequently graded in a blind setup alongside those of 200 participating students. We find that ChatGPT narrowly passed the exam, obtaining 20.5 out of 40 points. This impressive performance indicates that ChatGPT can indeed succeed in challenging tasks like university exams. At the same time, the questions in our exam are structurally similar to those of other exams, solved homework problems, and teaching materials that can be found online and might have been part of ChatGPT's training data. Therefore, it would be inadequate to conclude from this experiment that ChatGPT has any understanding of computer science. We also assess the improvements brought by GPT-4. We find that GPT-4 would have obtained about 17\% more exam points than GPT-3.5, reaching the performance of the average student. The transcripts of our conversations with ChatGPT are available at \url{https://github.com/tml-tuebingen/chatgpt-algorithm-exam}, and the entire graded exam is in the appendix of this paper.