Revisiting Aristotle vs. Ringelmann: The influence of biases on measuring productivity in Open Source software development

TL;DR

This paper investigates how sampling and instrumentation biases influence conclusions about whether open source project productivity scales sublinearly or superlinearly with team size, revealing biases significantly affect results.

Contribution

It provides a detailed comparison and replication of previous studies, identifying biases that impact the interpretation of productivity scaling in open source projects.

Findings

Sampling bias only partially explains discrepancies.

Instrumentation biases significantly influence regression outcomes.

Biases contribute more to differences than project population variations.

Abstract

Aristotle vs. Ringelmann was a discussion between two distinct research teams from the ETH Z\"urich who argued whether the productivity of Open Source software projects scales sublinear or superlinear with regard to its team size. This discussion evolved around two publications, which apparently used similar techniques by sampling projects on GitHub and running regression analyses to answer the question about superlinearity. Despite the similarity in their research methods, one team around Ingo Scholtes reached the conclusion that projects scale sublinear, while the other team around Didier Sornette ascertained a superlinear relationship between team size and productivity. In subsequent publications, the two authors argue that the opposite conclusions may be attributed to differences in project populations, since 81.7% of Sornette's projects have less than 50 contributors. Scholtes, on the other hand, sampled specifically projects with more than 50 contributors. This publication compares the research from both authors by replicating their findings, thus allowing for an evaluation of how much project sampling actually accounted for the differences between Scholtes' and Sornette's results. Thereby, the discovery was made that sampling bias only partially explains the discrepancies between the two authors. Further analysis led to the detection of instrumentation biases that drove the regression coefficients in opposite directions. These findings were then consolidated into a quantitative analysis, indicating that instrumentation biases contributed more to the differences between Scholtes' and Sornette's work than the selection bias suggested by both authors.