An Empirical Study on Usage and Perceptions of LLMs in a Software Engineering Project

TL;DR

This study investigates how software engineering students use and perceive large language models (LLMs) in academic projects, revealing their potential to improve early development stages and the importance of human-AI collaboration skills.

Contribution

It provides empirical insights into student interactions with LLMs in a real educational setting and offers a framework for effective LLM integration in software engineering education.

Findings

LLMs assist in generating foundational code structures

They help with syntax and error debugging

Students perceive LLMs as useful for early development stages

Abstract

Large Language Models (LLMs) represent a leap in artificial intelligence, excelling in tasks using human language(s). Although the main focus of general-purpose LLMs is not code generation, they have shown promising results in the domain. However, the usefulness of LLMs in an academic software engineering project has not been fully explored yet. In this study, we explore the usefulness of LLMs for 214 students working in teams consisting of up to six members. Notably, in the academic course through which this study is conducted, students were encouraged to integrate LLMs into their development tool-chain, in contrast to most other academic courses that explicitly prohibit the use of LLMs. In this paper, we analyze the AI-generated code, prompts used for code generation, and the human intervention levels to integrate the code into the code base. We also conduct a perception study to gain insights into the perceived usefulness, influencing factors, and future outlook of LLM from a computer science student's perspective. Our findings suggest that LLMs can play a crucial role in the early stages of software development, especially in generating foundational code structures, and helping with syntax and error debugging. These insights provide us with a framework on how to effectively utilize LLMs as a tool to enhance the productivity of software engineering students, and highlight the necessity of shifting the educational focus toward preparing students for successful human-AI collaboration.