How Far Are We? The Triumphs and Trials of Generative AI in Learning Software Engineering

TL;DR

This study evaluates ChatGPT's effectiveness in supporting software engineering students, finding no productivity gains but increased frustration and identifying interaction faults that impact user experience.

Contribution

It provides empirical insights into the current capabilities and challenges of conversational AI in software engineering education, highlighting interaction issues and user frustration.

Findings

No significant difference in productivity or self-efficacy

Increased frustration levels among students

Identification of five interaction faults causing negative outcomes

Abstract

Conversational Generative AI (convo-genAI) is revolutionizing Software Engineering (SE) as engineers and academics embrace this technology in their work. However, there is a gap in understanding the current potential and pitfalls of this technology, specifically in supporting students in SE tasks. In this work, we evaluate through a between-subjects study (N=22) the effectiveness of ChatGPT, a convo-genAI platform, in assisting students in SE tasks. Our study did not find statistical differences in participants' productivity or self-efficacy when using ChatGPT as compared to traditional resources, but we found significantly increased frustration levels. Our study also revealed 5 distinct faults arising from violations of Human-AI interaction guidelines, which led to 7 different (negative) consequences on participants.