Empirical Standards for Repository Mining

TL;DR

This paper discusses the development and presentation of empirical standards for repository mining to improve research quality and peer review reliability in software engineering.

Contribution

It introduces a specific empirical standard for mining software repositories, aiming to enhance research quality and peer review consistency.

Findings

Standard facilitates clearer communication of research expectations

Adoption of standards can improve research reliability

Provides a framework for community feedback and improvement

Abstract

The purpose of scholarly peer review is to evaluate the quality of scientific manuscripts. However, study after study demonstrates that peer review neither effectively nor reliably assesses research quality. Empirical standards attempt to address this problem by modelling a scientific community's expectations for each kind of empirical study conducted in that community. This should enhance not only the quality of research but also the reliability and predictability of peer review, as scientists adopt the standards in both their researcher and reviewer roles. However, these improvements depend on the quality and adoption of the standards. This tutorial will therefore present the empirical standard for mining software repositories, both to communicate its contents and to get feedback from the attendees. The tutorial will be organized into three parts: (1) brief overview of the empirical standards project; (2) detailed presentation of the repository mining standard; (3) discussion and suggestions for improvement.