If You Had to Pitch Your Ideal Software -- Evaluating Large Language Models to Support User Scenario Writing for User Experience Experts and Laypersons

TL;DR

This study evaluates how large language models can assist both UX experts and novices in writing user scenarios, revealing that LLMs enable laypersons to produce comparable quality descriptions to experts, especially in audience orientation.

Contribution

The paper demonstrates that LLM-supported tools can empower laypersons to create user scenarios of similar quality to experts, expanding requirements analysis capabilities.

Findings

LLMs help novices produce structured, clear user scenarios.

Laypersons' scenarios are comparable to experts in quality.

LLMs particularly improve audience-oriented descriptions.

Abstract

The process of requirements analysis requires an understanding of the end users of a system. Thus, expert stakeholders, such as User Experience (UX) designers, usually create various descriptions containing information about the users and their possible needs. In our paper, we investigate to what extent UX novices are able to write such descriptions into user scenarios. We conducted a user study with 60 participants consisting of 30 UX experts and 30 novices who were asked to write a user scenario with or without the help of an LLM-supported writing assistant. Our findings show that LLMs empower laypersons to write reasonable user scenarios and provide first-hand insights for requirements analysis that are comparable to UX experts in terms of structure and clarity, while especially excelling at audience-orientation. We present our qualitative and quantitative findings, including user scenario anatomies, potential influences, and differences in the way participants approached the task.