Is Our Organization Actually Measuring Productivity? How Contrasting Organizational and Individual Measures of Engineering Success is an Opportunity to Drive Engineering Transformation

TL;DR

This paper examines how organizations and individual engineers perceive and prioritize different measures of software engineering success, highlighting the importance of aligning metrics to improve engineering effectiveness.

Contribution

It introduces a Three Layer Productivity Framework and reveals misalignments between organizational and individual preferences for success metrics.

Findings

Participants favor increased focus on productivity metrics.

Organizations currently emphasize performance metrics.

Misalignment exists on specific metrics used for measuring success.

Abstract

Background: Previous research highlights that common misconceptions about developer productivity lead to harmful and inaccurate evaluations of software work, pointing to the need for organizations to differentiate between measures of production, productivity, and performance as an important step that helps to suggest improvements to how we measure the success of engineering teams. Methodology: Using a card sort, we explored how a Three Layer Productivity Framework was used by 16 software engineers at a Software Engineering focused conference to rank measures of success, first in the current practice of their organization and second in their individual beliefs about the best ways to measure engineering success. Results and discussion: Overall, participants preferred organizations to 1) continue their prioritized focus on performance layer metrics, 2) increase the focus on productivity metrics, and 3) decrease their focus on production metrics. When asked about the current metrics of their organizations, while all roles reported a current focus on performance metrics, only ICs reported a strong focus on production metrics. When asked about metrics they would prefer, all roles preferred more performance metrics but only leaders and ICs also wanted productivity metrics. While all participants were aligned on performance metrics being a top preference, there was misalignment on which specific metrics are used. Our findings show that when measuring developer success, organizations should continue measurement using performance metrics, consider an increased focus on productivity metrics, and consider a decreased focus on production metrics.