What Developers Ask to ChatGPT in GitHub Pull Requests? an Exploratory Study

TL;DR

This study explores how developers interact with ChatGPT in GitHub pull requests, categorizing request types and analyzing interaction patterns that lead to code contributions.

Contribution

It provides the first detailed categorization of ChatGPT requests in pull requests and analyzes interaction complexity and common request types.

Findings

14 types of ChatGPT requests identified

Code review and snippet implementation are common request types

Code generation prompts require more interactions

Abstract

The emergence of Large Language Models (LLMs), such as ChatGPT, has introduced a new set of tools to support software developers in solving pro- gramming tasks. However, our understanding of the interactions (i.e., prompts) between developers and ChatGPT that result in contributions to the codebase remains limited. To explore this limitation, we conducted a manual evaluation of 155 valid ChatGPT share links extracted from 139 merged Pull Requests (PRs), revealing the interactions between developers and reviewers with ChatGPT that led to merges into the main codebase. Our results produced a catalog of 14 types of ChatGPT requests categorized into four main groups. We found a significant number of requests involving code review and the implementation of code snippets based on specific tasks. Developers also sought to clarify doubts by requesting technical explanations or by asking for text refinements for their web pages. Furthermore, we verified that prompts involving code generation generally required more interactions to produce the desired answer compared to prompts requesting text review or technical information.