Evaluating Maintainability Prejudices with a Large-Scale Study of Open-Source Projects

TL;DR

This large-scale study empirically evaluates common prejudices about programming language maintainability using static analysis metrics across thousands of open-source projects, challenging many stereotypes.

Contribution

It provides the first extensive empirical validation of maintainability prejudices, revealing that many stereotypes lack support from real-world data.

Findings

JavaScript is not less maintainable than other languages.

Java code has higher maintainability than C#.

Code quality is not correlated with project popularity.

Abstract

Exaggeration or context changes can render maintainability experience into prejudice. For example, JavaScript is often seen as least elegant language and hence of lowest maintainability. Such prejudice should not guide decisions without prior empirical validation. We formulated 10 hypotheses about maintainability based on prejudices and test them in a large set of open-source projects (6,897 GitHub repositories, 402 million lines, 5 programming languages). We operationalize maintainability with five static analysis metrics. We found that JavaScript code is not worse than other code, Java code shows higher maintainability than C# code and C code has longer methods than other code. The quality of interface documentation is better in Java code than in other code. Code developed by teams is not of higher and large code bases not of lower maintainability. Projects with high maintainability are not more popular or more often forked. Overall, most hypotheses are not supported by open-source data.