From Bias To Improved Prompts: A Case Study of Bias Mitigation of Clone Detection Models

TL;DR

This paper evaluates large language models for clone code detection, identifies prompt bias as a key issue, and proposes a framework that improves detection accuracy by mitigating this bias.

Contribution

It introduces a novel framework to mitigate prompt bias in LLMs for clone detection, significantly enhancing model performance.

Findings

Palm model achieved high F1 scores of 89.30 and 86.41 on two datasets.

Identified eight categories of prompt bias affecting LLM performance.

Proposed bias mitigation approach improved F1 scores by up to 10.81%.

Abstract

The issue of clone code has persisted in software engineering, primarily because developers often copy and paste code segments. This common practice has elevated the importance of clone code detection, garnering attention from both software engineering researchers and industry professionals. Their collective concern arises from the potential negative impacts that clone code can have on software quality. The emergence of powerful Generative Large Language Models (LLMs) like ChatGPT has exacerbated the clone code problem. These advanced models possess code generation capabilities that can inadvertently create code clones. As a result, the need to detect clone code has become more critical than ever before. In this study, we assess the suitability of LLMs for clone code detection. Our results demonstrate that the Palm model achieved a high F1 score of 89.30 for the avatar dataset and 86.41 for the poolC dataset. A known issue with LLMs is their susceptibility to prompt bias, where the performance of these models fluctuates based on the input prompt provided. In our research, we delve deeper into the reasons behind these fluctuations and propose a framework to mitigate prompt bias for clone detection. Our analysis identifies eight distinct categories of prompt bias, and our devised approach leveraging these biases yields a significant improvement of up to 10.81% in the F1 score. These findings underscore the substantial impact of prompt bias on the performance of LLMs and highlight the potential for leveraging model errors to alleviate this bias.