Rebuttal to Berger et al., TOPLAS 2019

TL;DR

This paper is a rebuttal to critiques of a previous large-scale study on programming languages and code quality, confirming that the original findings are largely replicated and still valid.

Contribution

It defends the validity of the original study by showing that critiques largely replicate its results and confirms the robustness of its conclusions.

Findings

Results largely replicated by critics

Effects observed are small and should be interpreted with caution

Original conclusions remain valid and citable

Abstract

Berger et al., published in TOPLAS 2019, is a critique of our 2014 FSE conference abstract and its archival version, the 2017 CACM paper: A Large-Scale Study of Programming Languages and Code Quality in Github. In their paper Berger et al. make academic claims about the veracity of our work. Here, we respond to their technical and scientific critiques aimed at our work, attempting to stick with scientific discourse. We find that Berger et al. largely replicated our results, and agree with us in their conclusion: that the effects (in a statistical sense) found in the data are small, and should be taken with caution, and that it is possible that an absence of effect is the correct interpretation. Thus, our CACM paper's conclusions still hold, even more so now that they have been reproduced, and our paper is eminently citable.