The Impact of Generative AI on Code Expertise Models: An Exploratory Study

TL;DR

This study explores how generative AI tools like ChatGPT influence developer expertise models by analyzing their integration into code repositories and assessing the impact on expertise measurement accuracy.

Contribution

It provides an exploratory analysis of the effects of GenAI-generated code on source code knowledge models and expertise metrics, highlighting potential reliability issues.

Findings

GenAI integration affects expertise model accuracy

Most scenarios show measurable impact on knowledge metrics

Current expertise metrics are sensitive to AI-generated code

Abstract

Generative Artificial Intelligence (GenAI) tools for source code generation have significantly boosted productivity in software development. However, they also raise concerns, particularly the risk that developers may rely heavily on these tools, reducing their understanding of the generated code. We hypothesize that this loss of understanding may be reflected in source code knowledge models, which are used to identify developer expertise. In this work, we present an exploratory analysis of how a knowledge model and a Truck Factor algorithm built upon it can be affected by GenAI usage. To investigate this, we collected statistical data on the integration of ChatGPT-generated code into GitHub projects and simulated various scenarios by adjusting the degree of GenAI contribution. Our findings reveal that most scenarios led to measurable impacts, indicating the sensitivity of current expertise metrics. This suggests that as GenAI becomes more integrated into development workflows, the reliability of such metrics may decrease.