Impact of AI-tooling on the Engineering Workspace

TL;DR

This study analyzes how AI coding tools like Copilot affect engineering workflows across multiple companies, revealing changes in coding time, ticket size, PR process, and effort distribution, with benefits varying among companies.

Contribution

First multi-company analysis of AI tool impacts on engineering workflows, considering real-world settings and multiple indicators beyond productivity and satisfaction.

Findings

Coding time decreased by up to 15% among Copilot users.

Ticket sizes reduced by an average of 16%.

PR pickup times decreased by up to 33%.

Abstract

To understand the impacts of AI-driven coding tools on engineers' workflow and work environment, we utilize the Jellyfish platform to analyze indicators of change. Key indicators are derived from Allocations, Coding Fraction vs. PR Fraction, Lifecycle Phases, Cycle Time, Jira ticket size, PR pickup time, PR comments, PR comment count, interactions, and coding languages. Significant changes were observed in coding time fractions among Copilot users, with an average decrease of 3% with individual decreases as large as 15%. Ticket sizes decreased by an average of 16% across four companies, accompanied by an 8% decrease in cycle times, whereas the control group showed no change. Additionally, the PR process evolved with Copilot usage, featuring longer and more comprehensive comments, despite the weekly number of PRs reviewed remaining constant. Not all hypothesized changes were observed across all participating companies. However, some companies experienced a decrease in PR pickup times by up to 33%, indicating reduced workflow bottlenecks, and one company experienced a shift of up to 17% of effort from maintenance and support work towards product growth initiatives. This study is the first to utilize data from more than one company and goes beyond simple productivity and satisfaction measures, considering real-world engineering settings instead. By doing so, we highlight that some companies seem to benefit more than others from the use of Copilot and that changes can be subtle when investigating aggregates rather than specific aspects of engineering work and workflows - something that will be further investigated in the future.