An Empirical Study of LLM-Based Code Clone Detection

TL;DR

This study evaluates the performance and consistency of large language models in code clone detection across diverse datasets, revealing high performance on some datasets but significant variability on others, with most models showing consistent responses.

Contribution

It constructs multiple datasets from different sources and systematically evaluates LLMs, highlighting their strengths and limitations in code clone detection across datasets.

Findings

LLMs perform well on CodeNet datasets with high F1 scores

Performance drops significantly on BigCloneBench datasets

Most models exhibit over 90% response consistency across submissions

Abstract

Large language models (LLMs) have demonstrated remarkable capabilities in various software engineering tasks, such as code generation and debugging, because of their ability to translate between programming languages and natural languages. Existing studies have demonstrated the effectiveness of LLMs in code clone detection. However, two crucial issues remain unaddressed: the ability of LLMs to achieve comparable performance across different datasets and the consistency of LLMs' responses in code clone detection. To address these issues, we constructed seven code clone datasets and then evaluated five LLMs in four existing prompts with these datasets. The datasets were created by sampling code pairs using their Levenshtein ratio from two different code collections, CodeNet and BigCloneBench. Our evaluation revealed that although LLMs perform well in CodeNet-related datasets, with o3-mini achieving a 0.943 F1 score, their performance significantly decreased in BigCloneBench-related datasets. Most models achieved a high response consistency, with over 90\% of judgments remaining consistent across all five submissions. The fluctuations of the F1 score affected by inconsistency are also tiny; their variations are less than 0.03.