Exploring Creativity in Human-Human-LLM Collaborative Software Design

TL;DR

This study explores how Large Language Models (LLMs) influence creativity in collaborative software design, highlighting their supportive role alongside human designers in a simulated environment.

Contribution

It provides empirical insights into the ways LLMs support or hinder creativity during collaborative software design with experienced professionals.

Findings

Creativity was observed in all pairs, with 13 producing creative design documents.

LLMs contributed novel ideas and elaborated on human concepts.

In some cases, LLMs hindered creativity by suggesting complex solutions.

Abstract

While the use of Large Language Models (LLMs) in programming has been extensively studied, there is limited understanding of how LLMs support collaborative work where creativity plays a central role. Software design, as a collaborative and creative activity, provides a valuable context for exploring the influence of LLMs on creativity. This study investigates how and where creativity naturally emerges when software designers collaborate with an LLM during a design task. In a laboratory setting simulating a workplace environment, 18 pairs of software professionals with design experience were asked to complete a design task. Each pair had 90 minutes to produce a software design based on a set of requirements, with optional access to a custom LLM interface. Pairs were not primed to be creative. We find that creativity was present in all pairs in design processes, with 13 producing design documents containing creativity. We primarily attribute creativity to the human designers, driven by traits such as prior experience, empathy, and the use of analogies. The LLM contributed by producing novel ideas and elaborating human ideas. However, in some cases, the LLM appeared to hinder creativity by suggesting complex solutions or adding to unproductive digressions. LLMs can support creativity in collaborative software design, but human insights remain central. To effectively augment human creativity, designers must be intentional in their engagement with LLMs.