LecPrompt: A Prompt-based Approach for Logical Error Correction with CodeBERT

TL;DR

LecPrompt is a prompt-based method utilizing CodeBERT to detect and repair logical errors in code by analyzing perplexity metrics and framing correction as a Masked Language Modeling task, demonstrating high accuracy on Python and Java datasets.

Contribution

This paper introduces LecPrompt, a novel prompt-based approach that combines statistical analysis and MLM framing with CodeBERT for logical error detection and correction in code.

Findings

Achieves 74.58% token-level repair accuracy in Python

Attains 27.4% program-level repair accuracy in Python

Reaches 69.23% token-level repair accuracy in Java

Abstract

Logical errors in programming don't raise compiler alerts, making them hard to detect. These silent errors can disrupt a program's function or cause run-time issues. Their correction requires deep insight into the program's logic, highlighting the importance of automated detection and repair. In this paper, we introduce LecPrompt to localize and repair logical errors, an prompt-based approach that harnesses the capabilities of CodeBERT, a transformer-based large language model trained on code. First, LecPrompt leverages a large language model to calculate perplexity and log probability metrics, pinpointing logical errors at both token and line levels. Through statistical analysis, it identifies tokens and lines that deviate significantly from the expected patterns recognized by large language models, marking them as potential error sources. Second, by framing the logical error correction challenge as a Masked Language Modeling (MLM) task, LecPrompt employs CodeBERT to autoregressively repair the identified error tokens. Finally, the soft-prompt method provides a novel solution in low-cost scenarios, ensuring that the model can be fine-tuned to the specific nuances of the logical error correction task without incurring high computational costs. To evaluate LecPrompt's performance, we created a method to introduce logical errors into correct code and applying this on QuixBugs to produce the QuixBugs-LE dataset. Our evaluations on the QuixBugs-LE dataset for both Python and Java highlight the impressive capabilities of our method, LecPrompt. For Python, LecPrompt achieves a noteworthy 74.58% top-1 token-level repair accuracy and 27.4% program-level repair accuracy. In Java, LecPrompt delivers a 69.23\% top-1 token-level repair accuracy and 24.7% full program-level repair accuracy.