How the Misuse of a Dataset Harmed Semantic Clone Detection

TL;DR

This paper reveals that BigCloneBench, a widely used dataset for semantic clone detection evaluation, contains significant mislabeling issues that compromise the validity of many research results based on it.

Contribution

It provides a detailed analysis of BigCloneBench's flaws for semantic clone detection and highlights the impact on previous research and evaluation practices.

Findings

93% of sampled weak clone pairs are mislabelled with different functionality

139 out of 179 papers using BigCloneBench for semantic evaluation are affected by mislabelling

Misuse of BigCloneBench leads to overestimated F1 scores and misleading conclusions

Abstract

BigCloneBench is a well-known and widely used large-scale dataset for the evaluation of recall of clone detection tools. It has been beneficial for research on clone detection and has become a standard in evaluating the performance of clone detection tools. More recently, it has also been widely used as a dataset to evaluate machine learning approaches to semantic clone detection or code similarity detection for functional or semantic similarity. This paper demonstrates that BigCloneBench is problematic to use as ground truth for learning or evaluating semantic code similarity, and highlights the aspects of BigCloneBench that affect the ground truth quality. A manual investigation of a statistically significant random sample of 406 Weak Type-3/Type-4 clone pairs revealed that 93% of them do not have a similar functionality and are therefore mislabelled. In a literature review of 179 papers that use BigCloneBench as a dataset, we found 139 papers that used BigCloneBench to evaluate semantic clone detection and where the results are threatened in their validity by the mislabelling. As such, these papers often report high F1 scores (e.g., above 0.9), which indicates overfitting to dataset-specific artefacts rather than genuine semantic similarity detection. We emphasise that using BigCloneBench remains valid for the intended purpose of evaluating syntactic or textual clone detection of Type-1, Type-2, and Type-3 clones. We acknowledge the important contributions of BigCloneBench to two decades of traditional clone detection research. However, the usage of BigCloneBench beyond the intended purpose without careful consideration of its limitations has led to misleading results and conclusions, and potentially harmed the field of semantic clone detection.