The AI Companion in Education: Analyzing the Pedagogical Potential of ChatGPT in Computer Science and Engineering

TL;DR

This paper systematically evaluates ChatGPT's pedagogical potential in Computer Science and Engineering education, analyzing its strengths and limitations through diverse practice problems and a comprehensive reliability framework.

Contribution

It introduces a novel five-factor reliability analysis framework and provides insights into effective use cases and challenges of ChatGPT in CSE education.

Findings

ChatGPT performs well on conceptual knowledge questions.

Personalized and project-based tasks reveal more limitations.

Correlation analysis identifies key factors affecting response quality.

Abstract

Artificial Intelligence (AI), with ChatGPT as a prominent example, has recently taken center stage in various domains including higher education, particularly in Computer Science and Engineering (CSE). The AI revolution brings both convenience and controversy, offering substantial benefits while lacking formal guidance on their application. The primary objective of this work is to comprehensively analyze the pedagogical potential of ChatGPT in CSE education, understanding its strengths and limitations from the perspectives of educators and learners. We employ a systematic approach, creating a diverse range of educational practice problems within CSE field, focusing on various subjects such as data science, programming, AI, machine learning, networks, and more. According to our examinations, certain question types, like conceptual knowledge queries, typically do not pose significant challenges to ChatGPT, and thus, are excluded from our analysis. Alternatively, we focus our efforts on developing more in-depth and personalized questions and project-based tasks. These questions are presented to ChatGPT, followed by interactions to assess its effectiveness in delivering complete and meaningful responses. To this end, we propose a comprehensive five-factor reliability analysis framework to evaluate the responses. This assessment aims to identify when ChatGPT excels and when it faces challenges. Our study concludes with a correlation analysis, delving into the relationships among subjects, task types, and limiting factors. This analysis offers valuable insights to enhance ChatGPT's utility in CSE education, providing guidance to educators and students regarding its reliability and efficacy.