An exploratory analysis of Community-based Question-Answering Platforms and GPT-3-driven Generative AI: Is it the end of online community-based learning?

TL;DR

This study compares ChatGPT's answers to human solutions on Stack Overflow questions, revealing ChatGPT's comparable accuracy, shorter responses, and a decline in community engagement, raising questions about the future of online community learning.

Contribution

It provides an empirical analysis of ChatGPT's performance against human answers on Stack Overflow, highlighting its potential to influence online community-based learning.

Findings

ChatGPT answers are 66% shorter and 35% more similar to questions.

ChatGPT's answer accuracy is 71-75%.

Community comments on Stack Overflow decreased by 38%.

Abstract

Context: The advent of Large Language Model-driven tools like ChatGPT offers software engineers an interactive alternative to community question-answering (CQA) platforms like Stack Overflow. While Stack Overflow provides benefits from the accumulated crowd-sourced knowledge, it often suffers from unpleasant comments, reactions, and long waiting times. Objective: In this study, we assess the efficacy of ChatGPT in providing solutions to software engineering questions by analyzing its performance specifically against human solutions. Method: We empirically analyze 2564 Python and JavaScript questions from StackOverflow that were asked between January 2022 and December 2022. We parse the questions and answers from Stack Overflow, then collect the answers to the same questions from ChatGPT through API, and employ four textual and four cognitive metrics to compare the answers generated by ChatGPT with the accepted answers provided by human subject matter experts to find out the potential reasons for which future knowledge seekers may prefer ChatGPT over CQA platforms. We also measure the accuracy of the answers provided by ChatGPT. We also measure user interaction on StackOverflow over the past two years using three metrics to determine how ChatGPT affects it. Results: Our analysis indicates that ChatGPT's responses are 66% shorter and share 35% more words with the questions, showing a 25% increase in positive sentiment compared to human responses. ChatGPT's answers' accuracy rate is between 71 to 75%, with a variation in response characteristics between JavaScript and Python. Additionally, our findings suggest a recent 38% decrease in comment interactions on Stack Overflow, indicating a shift in community engagement patterns. A supplementary survey with 14 Python and JavaScript professionals validated these findings.