Is Stack Overflow Obsolete? An Empirical Study of the Characteristics of ChatGPT Answers to Stack Overflow Questions

TL;DR

This study analyzes ChatGPT's answers to Stack Overflow questions, revealing issues with correctness and verbosity, while also noting user preferences for its comprehensive and articulate responses despite misinformation risks.

Contribution

First comprehensive empirical analysis of ChatGPT answers to programming questions, highlighting their strengths and weaknesses compared to traditional Q&A platforms.

Findings

52% of ChatGPT answers contain incorrect information

77% of answers are verbose

Users prefer ChatGPT answers 35% of the time despite misinformation

Abstract

Q&A platforms have been crucial for the online help-seeking behavior of programmers. However, the recent popularity of ChatGPT is altering this trend. Despite this popularity, no comprehensive study has been conducted to evaluate the characteristics of ChatGPT's answers to programming questions. To bridge the gap, we conducted the first in-depth analysis of ChatGPT answers to 517 programming questions on Stack Overflow and examined the correctness, consistency, comprehensiveness, and conciseness of ChatGPT answers. Furthermore, we conducted a large-scale linguistic analysis, as well as a user study, to understand the characteristics of ChatGPT answers from linguistic and human aspects. Our analysis shows that 52% of ChatGPT answers contain incorrect information and 77% are verbose. Nonetheless, our user study participants still preferred ChatGPT answers 35% of the time due to their comprehensiveness and well-articulated language style. However, they also overlooked the misinformation in the ChatGPT answers 39% of the time. This implies the need to counter misinformation in ChatGPT answers to programming questions and raise awareness of the risks associated with seemingly correct answers.