ChatGPT: Friend or Foe When Comprehending and Changing Unfamiliar Code

TL;DR

This study explores how AI tools influence the problem-solving behaviors of developers when understanding and modifying unfamiliar code, revealing nuanced impacts on their cognitive processes and stuckness.

Contribution

It provides an empirical analysis of AI's effects on developers' problem-solving approaches and stuckness during complex coding tasks, using Polya's problem-solving phases.

Findings

AI tools are repeatedly used to offload tasks but do not change core problem-solving behaviors.

Most developers experienced being stuck, with AI sometimes helping and sometimes hindering their progress.

Seven distinct causes for being stuck were identified, with AI impacting these in varied ways.

Abstract

A rapidly growing body of research is examining how LLMs influence developers when they code. To date, this research has tended to focus on productivity and code quality outcomes, rather than the underlying cognitive processes involved in programming. To address this gap, we report on the results of an exploratory laboratory study of ten advanced student developers (five with support from AI and five without) who had to make a non-trivial extension to a sizable software system. Leveraging Polya's four problem-solving phases and 25 inductively-generated codes detailing distinct problem-solving behaviors as the primary lenses, we examined: (1) how AI impacted the problem-solving approach the developers used to solve the programming task, and (2) how AI impacted their progress when they became stuck. For the analysis, we triangulated data across multiple sources (e.g., think-aloud, code changes, web searches, and LLM prompts). Unexpectedly, while developers in the AI group repeatedly turned to the AI tool to offload certain aspects of the process, all detailed problem-solving behaviors appeared in both groups. We also found that nine out of ten participants found themselves stuck in their work, but with key differences in how they became stuck and unstuck. We highlight seven distinct causes for being stuck and highlight how AI in some cases helped and in other cases hindered becoming unstuck.