Impact of the Availability of ChatGPT on Software Development: A Synthetic Difference in Differences Estimation using GitHub Data

TL;DR

This study assesses how ChatGPT influences software development productivity using GitHub data and advanced causal inference methods, revealing significant positive effects on developer activity and repository creation, especially in certain programming languages.

Contribution

It introduces a novel application of SDID to measure ChatGPT's impact on software development using GitHub data and natural experiments.

Findings

Increased git pushes, repositories, and developers per 100,000 people.

Significant positive effects in high-level and scripting languages.

Potential concerns include code quality and privacy issues.

Abstract

Advancements in Artificial Intelligence, particularly with ChatGPT, have significantly impacted software development. Utilizing novel data from GitHub Innovation Graph, we hypothesize that ChatGPT enhances software production efficiency. Utilizing natural experiments where some governments banned ChatGPT, we employ Difference-in-Differences (DID), Synthetic Control (SC), and Synthetic Difference-in-Differences (SDID) methods to estimate its effects. Our findings indicate a significant positive impact on the number of git pushes, repositories, and unique developers per 100,000 people, particularly for high-level, general purpose, and shell scripting languages. These results suggest that AI tools like ChatGPT can substantially boost developer productivity, though further analysis is needed to address potential downsides such as low quality code and privacy concerns.