Publication Bias: A Detailed Analysis of Experiments Published in ESEM

TL;DR

This study investigates publication bias in experiments published in ESEM, finding limited evidence of bias and suggesting that SE research practices are relatively unbiased compared to other fields.

Contribution

The paper provides a detailed analysis of publication bias in ESEM experiments and offers insights into research practices that may influence bias.

Findings

ESEM experiments rarely define hypotheses or perform multiple testing.

One-tailed tests are slightly more likely to yield significant results.

No strong evidence of publication bias was found in ESEM publications.

Abstract

Background: Publication bias is the failure to publish the results of a study based on the direction or strength of the study findings. The existence of publication bias is firmly established in areas like medical research. Recent research suggests the existence of publication bias in Software Engineering. Aims: Finding out whether experiments published in the International Workshop on Empirical Software Engineering and Measurement (ESEM) are affected by publication bias. Method: We review experiments published in ESEM. We also survey with experimental researchers to triangulate our findings. Results: ESEM experiments do not define hypotheses and frequently perform multiple testing. One-tailed tests have a slightly higher rate of achieving statistically significant results. We could not find other practices associated with publication bias. Conclusions: Our results provide a more encouraging perspective of SE research than previous research: (1) ESEM publications do not seem to be strongly affected by biases and (2) we identify some practices that could be associated with p-hacking, but it is more likely that they are related to the conduction of exploratory research.