The Role of LLMs in Collaborative Software Design

TL;DR

This study explores how large language models influence collaborative software design, revealing varied usage patterns, impacts on understanding, and implications for tool development in professional settings.

Contribution

It provides empirical insights into the diverse ways professionals use LLMs collaboratively in software design, highlighting effects on understanding and exploration.

Findings

Shared-instance use improves shared understanding.

Parallel use can cause context drift.

Professionals critically evaluate LLM responses, influencing design insights.

Abstract

While much prior work examines Large Language Models (LLMs) for solo development tasks (e.g., coding), far less is known about how LLMs shape collaborative group work in software engineering. This study focuses on one such collaborative task, namely software design. It presents the results of an exploratory laboratory study of 18 pairs of software professionals who could use an LLM however they saw fit, to design a University campus bicycle parking application. Our findings reveal that introducing an LLM leads to distinct patterns of joint use: shared-instance use facilitated shared understanding, whereas parallel use across separate instances sometimes led to ''context drift''. We also observe wide variation in reliance, from non-use to treating the LLM as an information source or producer. Across these modes, professionals scrutinized and reflected on LLM responses, often yielding design insights; however, early anchoring sometimes curtailed exploration. We provide implications for tools to aid designers while retaining the human-centricity important to design.