From GPT-3 to GPT-4: On the Evolving Efficacy of LLMs to Answer Multiple-choice Questions for Programming Classes in Higher Education

TL;DR

This paper analyzes how the capabilities of GPT models in answering programming multiple-choice questions have evolved from before ChatGPT's release to August 2023, revealing increasing efficacy and implications for educational assessments.

Contribution

It provides a comparative evaluation of three GPT models over time on Python MCQs, highlighting qualitative differences and informing assessment design in programming education.

Findings

GPT models' ability to answer MCQs has significantly improved over time.

Recent GPT models can pass current programming assessments with minimal effort.

The study offers insights for educators to adapt assessment strategies based on AI capabilities.

Abstract

We explore the evolving efficacy of three generative pre-trained transformer (GPT) models in generating answers for multiple-choice questions (MCQ) from introductory and intermediate Python programming courses in higher education. We focus on the differences in capabilities of the models prior to the release of ChatGPT (Nov '22), at the time of the release, and today (i.e., Aug '23). Recent studies have established that the abilities of the OpenAI's GPT models to handle assessments originally designed for humans keep increasing as the newer more capable models are released. However, the qualitative differences in the capabilities and limitations of these models to reason about and/or analyze programming MCQs have been under-explored. We evaluated three OpenAI's GPT models on formative and summative MCQ assessments from three Python courses (530 questions) focusing on the qualitative differences in the evolving efficacy of the subsequent models. This study provides further evidence and insight into the trajectory of the current developments where there already exists a technology that can be utilized by students to collect passing scores, with no effort whatsoever, on what today counts as viable programming knowledge and skills assessments. This study could be leveraged by educators and institutions to better understand the recent technological developments in order to adapt the design of programming assessments as well as to fuel the necessary discussions into how assessments in future programming classes should be updated.