Can Developers Prompt? A Controlled Experiment for Code Documentation Generation
Hans-Alexander Kruse, Tim Puhlf\"ur{\ss}, Walid Maalej

TL;DR
This study investigates how developers and students use prompt engineering with LLMs like ChatGPT for code documentation, revealing limited awareness of techniques and highlighting the need for better support and training.
Contribution
It provides empirical insights into developer behaviors and perceptions when using LLMs for code documentation, emphasizing the gap in prompt engineering skills.
Findings
Students found ad-hoc prompts less effective than predefined ones.
Professionals sometimes improved output by including specific keywords.
Participants viewed LLMs as iterative tools rather than perfect solutions.
Abstract
Large language models (LLMs) bear great potential for automating tedious development tasks such as creating and maintaining code documentation. However, it is unclear to what extent developers can effectively prompt LLMs to create concise and useful documentation. We report on a controlled experiment with 20 professionals and 30 computer science students tasked with code documentation generation for two Python functions. The experimental group freely entered ad-hoc prompts in a ChatGPT-like extension of Visual Studio Code, while the control group executed a predefined few-shot prompt. Our results reveal that professionals and students were unaware of or unable to apply prompt engineering techniques. Especially students perceived the documentation produced from ad-hoc prompts as significantly less readable, less concise, and less helpful than documentation from prepared prompts. Some…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsModel-Driven Software Engineering Techniques · Software Engineering Research · Advanced Software Engineering Methodologies
