What Characteristics Make ChatGPT Effective for Software Issue Resolution? An Empirical Study of Task, Project, and Conversational Signals in GitHub Issues
Ramtin Ehsani, Sakshi Pathak, Esteban Parra, Sonia Haiduc, Preetha Chatterjee

TL;DR
This study analyzes 686 GitHub developer-ChatGPT conversations to identify characteristics that make LLM interactions effective for software issue resolution, highlighting factors like task type, project size, and conversation quality.
Contribution
It provides an empirical analysis of what makes ChatGPT conversations helpful in resolving software issues, offering insights for improving developer tools and LLM fine-tuning.
Findings
ChatGPT is most effective for code generation and API recommendations.
Helpful conversations are shorter, more readable, and semantically aligned.
Larger projects and experienced developers benefit more from ChatGPT.
Abstract
Conversational large-language models are extensively used for issue resolution tasks. However, not all developer-LLM conversations are useful for effective issue resolution. In this paper, we analyze 686 developer-ChatGPT conversations shared within GitHub issue threads to identify characteristics that make these conversations effective for issue resolution. First, we analyze the conversations and their corresponding issues to distinguish helpful from unhelpful conversations. We begin by categorizing the types of tasks developers seek help with to better understand the scenarios in which ChatGPT is most effective. Next, we examine a wide range of conversational, project, and issue-related metrics to uncover factors associated with helpful conversations. Finally, we identify common deficiencies in unhelpful ChatGPT responses to highlight areas that could inform the design of more…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
