"Always Nice and Confident, Sometimes Wrong": Developer's Experiences Engaging Large Language Models (LLMs) Versus Human-Powered Q&A Platforms for Coding Support

TL;DR

This study compares human-powered Q&A platforms like Stack Overflow with AI chatbots such as ChatGPT, highlighting their respective strengths, limitations, and implications for future developer support tools based on thematic analysis of Reddit discussions.

Contribution

It provides a comparative analysis of Stack Overflow and ChatGPT in real-world coding support, revealing their unique advantages and challenges for developers.

Findings

ChatGPT offers fast, clear, and comprehensive responses.

ChatGPT fosters a more respectful environment than SO.

Concerns exist about ChatGPT's reliability due to overconfidence.

Abstract

Software engineers have historically relied on human-powered Q&A platforms like Stack Overflow (SO) as coding aids. With the rise of generative AI, developers have started to adopt AI chatbots, such as ChatGPT, in their software development process. Recognizing the potential parallels between human-powered Q&A platforms and AI-powered question-based chatbots, we investigate and compare how developers integrate this assistance into their real-world coding experiences by conducting a thematic analysis of 1700+ Reddit posts. Through a comparative study of SO and ChatGPT, we identified each platform's strengths, use cases, and barriers. Our findings suggest that ChatGPT offers fast, clear, comprehensive responses and fosters a more respectful environment than SO. However, concerns about ChatGPT's reliability stem from its overly confident tone and the absence of validation mechanisms like SO's voting system. Based on these findings, we synthesized the design implications for future GenAI code assistants and recommend a workflow leveraging each platform's unique features to improve developer experiences.