SourcererCC: Scaling Code Clone Detection to Big Code

TL;DR

SourcererCC is a scalable, token-based code clone detection tool capable of analyzing large code repositories efficiently while maintaining high recall and precision.

Contribution

It introduces an optimized index and filtering heuristics enabling clone detection across large repositories on standard workstations.

Findings

Achieves high recall and precision in clone detection.

Scales to repositories with 250 million lines of code.

Outperforms four state-of-the-art tools in evaluation.

Abstract

Despite a decade of active research, there is a marked lack in clone detectors that scale to very large repositories of source code, in particular for detecting near-miss clones where significant editing activities may take place in the cloned code. We present SourcererCC, a token-based clone detector that targets three clone types, and exploits an index to achieve scalability to large inter-project repositories using a standard workstation. SourcererCC uses an optimized inverted-index to quickly query the potential clones of a given code block. Filtering heuristics based on token ordering are used to significantly reduce the size of the index, the number of code-block comparisons needed to detect the clones, as well as the number of required token-comparisons needed to judge a potential clone. We evaluate the scalability, execution time, recall and precision of SourcererCC, and compare it to four publicly available and state-of-the-art tools. To measure recall, we use two recent benchmarks, (1) a large benchmark of real clones, BigCloneBench, and (2) a Mutation/Injection-based framework of thousands of fine-grained artificial clones. We find SourcererCC has both high recall and precision, and is able to scale to a large inter-project repository (250MLOC) using a standard workstation.