Does Co-Development with AI Assistants Lead to More Maintainable Code? A Registered Report

TL;DR

This study investigates whether AI assistants like GitHub Copilot improve software maintainability by assessing their impact on developers' ability to evolve code through controlled experiments.

Contribution

It introduces a two-phased experimental design to systematically evaluate AI's effect on code maintainability and developer productivity.

Findings

Anticipated insights into AI's influence on code evolution.

Evaluation of developer productivity with and without AI.

Assessment of code quality and test coverage differences.

Abstract

[Background/Context] AI assistants like GitHub Copilot are transforming software engineering; several studies have highlighted productivity improvements. However, their impact on code quality, particularly in terms of maintainability, requires further investigation. [Objective/Aim] This study aims to examine the influence of AI assistants on software maintainability, specifically assessing how these tools affect the ability of developers to evolve code. [Method] We will conduct a two-phased controlled experiment involving professional developers. In Phase 1, developers will add a new feature to a Java project, with or without the aid of an AI assistant. Phase 2, a randomized controlled trial, will involve a different set of developers evolving random Phase 1 projects - working without AI assistants. We will employ Bayesian analysis to evaluate differences in completion time, perceived productivity, code quality, and test coverage.