Unveiling the Role of ChatGPT in Software Development: Insights from Developer-ChatGPT Interactions on GitHub

TL;DR

This study provides an empirical analysis of how developers use ChatGPT in real-world software development, revealing interaction patterns, purposes, and key activities supported by ChatGPT on GitHub.

Contribution

It introduces DevChat, a large dataset of developer-ChatGPT interactions, and maps ChatGPT's practical roles and tasks in software development.

Findings

Interactions are typically short and task-focused

Developers mainly share conversations to delegate tasks and resolve problems

ChatGPT is frequently used for software implementation and maintenance

Abstract

The advent of Large Language Models (LLMs) has introduced a new paradigm in Software Engineering (SE), with generative AI tools like ChatGPT gaining widespread adoption among developers. While ChatGPT's potential has been extensively discussed, empirical evidence about how developers actually use LLMs' assistance in real-world practices remains limited. To bridge this gap, we conducted a large-scale empirical analysis of ChatGPT usage on GitHub, and we presented DevChat, a curated dataset of 2,547 publicly shared ChatGPT conversation links collected from GitHub between May 2023 and June 2024. Through comprehensively analyzing DevChat, we explored the characteristics of developer-ChatGPT interaction patterns and identified five key categories of developers' purposes for sharing developer-ChatGPT conversations during software development. Additionally, we investigated the dominant development-related activities in which ChatGPT is used, and presented a mapping framework that links GitHub data sources, development-related activities, and SE tasks. The findings show that interactions are typically short and task-focused (most are 1-3 turns); developers share conversations mainly to delegate tasks, resolve problems, and acquire knowledge, revealing five purpose categories; ChatGPT is most frequently engaged for Software Implementation and Maintenance & Evolution; we identified 39 fine-grained SE tasks supported by ChatGPT, with Code Generation & Completion as well as Code modification & Optimization being the most prominent. Our study offers a comprehensive mapping of ChatGPT's applications in real-world software development scenarios and provides a foundation for understanding LLMs' practical roles in software development.