Can Developers Prompt? A Controlled Experiment for Code Documentation Generation

TL;DR

This study investigates how developers and students use prompt engineering with LLMs like ChatGPT for code documentation, revealing limited awareness of techniques and highlighting the need for better support and training.

Contribution

It provides empirical insights into developer behaviors and perceptions when using LLMs for code documentation, emphasizing the gap in prompt engineering skills.

Findings

Students found ad-hoc prompts less effective than predefined ones.

Professionals sometimes improved output by including specific keywords.

Participants viewed LLMs as iterative tools rather than perfect solutions.

Abstract

Large language models (LLMs) bear great potential for automating tedious development tasks such as creating and maintaining code documentation. However, it is unclear to what extent developers can effectively prompt LLMs to create concise and useful documentation. We report on a controlled experiment with 20 professionals and 30 computer science students tasked with code documentation generation for two Python functions. The experimental group freely entered ad-hoc prompts in a ChatGPT-like extension of Visual Studio Code, while the control group executed a predefined few-shot prompt. Our results reveal that professionals and students were unaware of or unable to apply prompt engineering techniques. Especially students perceived the documentation produced from ad-hoc prompts as significantly less readable, less concise, and less helpful than documentation from prepared prompts. Some professionals produced higher quality documentation by just including the keyword Docstring in their ad-hoc prompts. While students desired more support in formulating prompts, professionals appreciated the flexibility of ad-hoc prompting. Participants in both groups rarely assessed the output as perfect. Instead, they understood the tools as support to iteratively refine the documentation. Further research is needed to understand which prompting skills and preferences developers have and which support they need for certain tasks.