Fine-Tuning Code Language Models to Detect Cross-Language Bugs

TL;DR

This paper explores the use of fine-tuned code language models for detecting cross-language bugs in multilingual programming, demonstrating that targeted fine-tuning improves detection accuracy.

Contribution

It introduces CLCFinder, a dataset for cross-language bugs, and shows that fine-tuning CodeLMs enhances their ability to detect these bugs, especially with larger datasets.

Findings

All 13 CodeLMs improved after fine-tuning.

Small CodeLMs sometimes outperform larger ones.

Increasing dataset size boosts performance.

Abstract

Multilingual programming, which involves using multiple programming languages (PLs) in a single project, is increasingly common due to its benefits. However, it introduces cross-language bugs (CLBs), which arise from interactions between different PLs and are difficult to detect by single-language bug detection tools. This paper investigates the potential of pre-trained code language models (CodeLMs) in CLB detection. We developed CLCFinder, a cross-language code identification tool, and constructed a CLB dataset involving three PL combinations (Python-C/C++, Java-C/C++, and Python-Java) with nine interaction types. We fine-tuned 13 CodeLMs on this dataset and evaluated their performance, analyzing the effects of dataset size, token sequence length, and code comments. Results show that all 13 CodeLMs exhibited varying degrees of performance improvement after fine-tuning, with UniXcoder-base achieving the best F1 score (0.7407). Notably, within our experimental setup, small CodeLMs tended to performe better than large ones. CodeLMs fine-tuned on single-language bug datasets performed poorly on CLB detection, demonstrating the distinction between CLBs and single-language bugs. Additionally, increasing the fine-tuning dataset size significantly improved performance, while longer token sequences did not necessarily improve the model performance. The impact of code comments varied across models. Some fine-tuned CodeLMs' performance was improved, while others showed degraded performance.