The potential of large language models for improving probability learning: A study on ChatGPT3.5 and first-year computer engineering students

TL;DR

This study evaluates ChatGPT 3.5's effectiveness in solving introductory probability problems for computer engineering students, highlighting its strengths in reasoning and explanation, but noting limitations in numerical calculations.

Contribution

It demonstrates that large language models like ChatGPT can outperform students in probability problem-solving and serve as potential learning assistants despite some computational limitations.

Findings

ChatGPT surpasses average students in probability reasoning and explanation.

The model performs consistently across languages.

Using R scripts improves numerical task performance.

Abstract

In this paper, we assess the efficacy of ChatGPT (version Feb 2023), a large-scale language model, in solving probability problems typically presented in introductory computer engineering exams. Our study comprised a set of 23 probability exercises administered to students at Rey Juan Carlos University (URJC) in Madrid. The responses produced by ChatGPT were evaluated by a group of five statistics professors, who assessed them qualitatively and assigned grades based on the same criteria used for students. Our results indicate that ChatGPT surpasses the average student in terms of phrasing, organization, and logical reasoning. The model's performance remained consistent for both the Spanish and English versions of the exercises. However, ChatGPT encountered difficulties in executing basic numerical operations. Our experiments demonstrate that requesting ChatGPT to provide the solution in the form of an R script proved to be an effective approach for overcoming these limitations. In summary, our results indicate that ChatGPT surpasses the average student in solving probability problems commonly presented in introductory computer engineering exams. Nonetheless, the model exhibits limitations in reasoning around certain probability concepts. The model's ability to deliver high-quality explanations and illustrate solutions in any programming language, coupled with its performance in solving probability exercises, suggests that large language models have the potential to serve as learning assistants.