Technical Debt and Maintainability: How do tools measure it?

TL;DR

This paper systematically analyzes 11 tools used to measure technical debt and maintainability, revealing their focus on internal code qualities, diverse definitions, and the opaqueness of their measurements.

Contribution

It provides an in-depth comparison of existing tools for assessing technical debt and maintainability, highlighting their differences and limitations.

Findings

Tools mainly focus on internal code quality.

Measurements are often opaque and inconsistent.

Trustworthiness of tools varies widely.

Abstract

The technical state of software, i.e., its technical debt (TD) and maintainability are of increasing interest as ever more software is developed and deployed. Since td and maintainability are neither uniformly defined, not easy to understand, nor directly measurable, practitioners are likely to apply readily available tools to assess TD or maintainability and they may rely on the reported results without properly understanding what they embody. In this paper, we: a) methodically identify 11 readily available tools that measure TD or maintainability, b) present an in-depth investigation on how each of these tools measures and computes TD or maintainability, and c) compare these tools and their characteristics. We find that contemporary tools focus mainly on internal qualities of software, i.e., quality of source code, that they define and measure TD or maintainability in widely different ways, that most of the tools measure TD or maintainability opaquely, and that it is not obvious why the measure of one tool is more trustworthy or representative than the one of another.