Exploring LLMs Impact on Student-Created User Stories and Acceptance Testing in Software Development

TL;DR

This study examines how large language models influence undergraduate students' ability to craft user stories and acceptance criteria in Agile development, highlighting benefits and limitations of LLM assistance.

Contribution

It demonstrates the impact of guided LLMs on student-generated user stories, revealing both advantages and challenges in their use for educational purposes.

Findings

LLMs help students create well-defined acceptance criteria.

Students perform better without LLMs when focusing on appropriate scope.

LLMs assist in developing valuable user stories.

Abstract

In Agile software development methodology, a user story describes a new feature or functionality from an end user's perspective. The user story details may also incorporate acceptance testing criteria, which can be developed through negotiation with users. When creating stories from user feedback, the software engineer may maximize their usefulness by considering story attributes, including scope, independence, negotiability, and testability. This study investigates how LLMs (large language models), with guided instructions, affect undergraduate software engineering students' ability to transform user feedback into user stories. Students, working individually, were asked to analyze user feedback comments, appropriately group related items, and create user stories following the principles of INVEST, a framework for assessing user stories. We found that LLMs help students develop valuable stories with well-defined acceptance criteria. However, students tend to perform better without LLMs when creating user stories with an appropriate scope.