Beyond Code Generation: An Observational Study of ChatGPT Usage in Software Engineering Practice

TL;DR

This study observes how 24 professional software engineers use ChatGPT over a week, revealing they mainly seek guidance and learning rather than ready-made code, and introduces a framework for understanding their experience.

Contribution

It provides empirical insights into real-world ChatGPT usage in industry and proposes a theoretical framework for factors influencing user experience.

Findings

Engineers use ChatGPT mainly for guidance and learning.

ChatGPT is less used for generating ready-to-use code.

A framework links purpose, personality, and external factors to perceived usefulness.

Abstract

Large Language Models (LLMs) are frequently discussed in academia and the general public as support tools for virtually any use case that relies on the production of text, including software engineering. Currently there is much debate, but little empirical evidence, regarding the practical usefulness of LLM-based tools such as ChatGPT for engineers in industry. We conduct an observational study of 24 professional software engineers who have been using ChatGPT over a period of one week in their jobs, and qualitatively analyse their dialogues with the chatbot as well as their overall experience (as captured by an exit survey). We find that, rather than expecting ChatGPT to generate ready-to-use software artifacts (e.g., code), practitioners more often use ChatGPT to receive guidance on how to solve their tasks or learn about a topic in more abstract terms. We also propose a theoretical framework for how (i) purpose of the interaction, (ii) internal factors (e.g., the user's personality), and (iii) external factors (e.g., company policy) together shape the experience (in terms of perceived usefulness and trust). We envision that our framework can be used by future research to further the academic discussion on LLM usage by software engineering practitioners, and to serve as a reference point for the design of future empirical LLM research in this domain.