"Like Taking the Path of Least Resistance": Exploring the Impact of LLM Interaction on the Creative Process of Programming

TL;DR

This study examines how large language models influence programmers' creative processes, revealing that LLMs shorten idea generation, alter collaboration modes, and produce solutions with similar idea diversity but higher correctness.

Contribution

It provides empirical insights into human-LLM collaboration modes and their effects on programming creativity, highlighting both benefits and limitations.

Findings

LLMs significantly shorten idea-generation time (p=0.0004)

Participants had fewer creative moments with LLM assistance (p=0.002)

LLMs produce more correct and functional code but similar idea diversity

Abstract

Creativity is fundamentally human. As AI takes on more of the generative work that once required human imagination, despite documented limitations in creative ability, a critical question emerges: How does GenAI affect users' creativity? Through a within-subject study followed by retrospective interviews with (N=20) programmers, we investigated the impact of LLMs on participants' process of creative thinking in programming and the creativity of generated solutions. Across two conditions (LLM-assisted vs. unassisted), participants using LLMs had significantly shorter idea-generation periods (p=0.0004), leading to fewer creative moments (p=0.002). Qualitative analysis of participants' interactions and interviews revealed four different human-LLM collaboration modes supporting various problem-solving strategies. However, a comparative analysis of the generated solutions shows that while LLMs can help generate more correct and functional code, their solutions contain roughly the same number of ideas as participant-generated ones. Based on our findings, we discuss design implications and considerations for effectively using LLMs to support user creativity.