Comparing Software Developers with ChatGPT: An Empirical Investigation

TL;DR

This paper empirically compares the performance of software engineers and ChatGPT across various tasks and metrics to evaluate AI's role and effectiveness in software development.

Contribution

It provides the first comprehensive empirical comparison between human developers and ChatGPT, considering multiple evaluation criteria for collaborative software engineering.

Findings

ChatGPT-generated code performs comparably to human code in some tasks.

AI systems like ChatGPT show potential to assist in software development.

The study highlights areas where AI can improve human-AI collaboration.

Abstract

The advent of automation in particular Software Engineering (SE) tasks has transitioned from theory to reality. Numerous scholarly articles have documented the successful application of Artificial Intelligence to address issues in areas such as project management, modeling, testing, and development. A recent innovation is the introduction of ChatGPT, an ML-infused chatbot, touted as a resource proficient in generating programming codes and formulating software testing strategies for developers and testers respectively. Although there is speculation that AI-based computation can increase productivity and even substitute software engineers in software development, there is currently a lack of empirical evidence to verify this. Moreover, despite the primary focus on enhancing the accuracy of AI systems, non-functional requirements including energy efficiency, vulnerability, fairness (i.e., human bias), and safety frequently receive insufficient attention. This paper posits that a comprehensive comparison of software engineers and AI-based solutions, considering various evaluation criteria, is pivotal in fostering human-machine collaboration, enhancing the reliability of AI-based methods, and understanding task suitability for humans or AI. Furthermore, it facilitates the effective implementation of cooperative work structures and human-in-the-loop processes. This paper conducts an empirical investigation, contrasting the performance of software engineers and AI systems, like ChatGPT, across different evaluation metrics. The empirical study includes a case of assessing ChatGPT-generated code versus code produced by developers and uploaded in Leetcode.