ChatGPT and Its Educational Impact: Insights from a Software Development Competition

TL;DR

This paper investigates how ChatGPT integration in a student software development competition improves project outcomes, learning experiences, and satisfaction, demonstrating its potential to transform educational practices.

Contribution

It presents the first empirical evaluation of ChatGPT's impact on student project performance and learning in a competitive educational setting.

Findings

Students using ChatGPT achieved higher project scores.

ChatGPT use correlated with increased project completion rates.

Participants reported high satisfaction with ChatGPT's role in learning.

Abstract

This study explores the integration and impact of ChatGPT, a generative AI that utilizes natural language processing, in an educational environment. The main goal is to evaluate how ChatGPT affects project performance. To this end, we organize a software development competition utilizing ChatGPT, lasting for four weeks and involving 36 students. The competition is structured in two rounds: in the first round, all 36 students participate and are evaluated based on specific performance metrics such as code quality, innovation, and adherence to project requirements. The top 15 performers from the first round are then selected to advance to the second round, where they compete for the final rankings and the overall winner is determined. The competition shows that students who use ChatGPT extensively in various stages of development, including ideation, documentation, software development, and quality assurance, have higher project completion rates and better scores. A detailed comparative analysis between first-round and second-round winners reveals significant differences in their experience with generative AI for software development, experience learning large-scale language models, and interest in their respective fields of study. These findings suggest that ChatGPT enhances individual learning and project performance. A post-survey of participants also reveals high levels of satisfaction, further emphasizing the benefits of integrating generative AI like ChatGPT in academic settings. This study highlights the transformative potential of ChatGPT in project-based learning environments and supports further research into its long-term impact and broader application in a variety of educational contexts.