Uncovering LLM-Generated Code: A Zero-Shot Synthetic Code Detector via Code Rewriting

TL;DR

This paper introduces a zero-shot method for detecting LLM-generated code by analyzing the similarity between original and rewritten code, significantly outperforming existing detectors on benchmark datasets.

Contribution

It proposes a novel zero-shot synthetic code detection approach using code rewriting and contrastive learning, addressing the limitations of general text detectors for programming code.

Findings

Achieved 20.5% AUROC improvement on APPS benchmark.

Achieved 29.1% AUROC improvement on MBPP benchmark.

Outperforms existing state-of-the-art synthetic code detectors.

Abstract

Large Language Models (LLMs) have demonstrated remarkable proficiency in generating code. However, the misuse of LLM-generated (synthetic) code has raised concerns in both educational and industrial contexts, underscoring the urgent need for synthetic code detectors. Existing methods for detecting synthetic content are primarily designed for general text and struggle with code due to the unique grammatical structure of programming languages and the presence of numerous ''low-entropy'' tokens. Building on this, our work proposes a novel zero-shot synthetic code detector based on the similarity between the original code and its LLM-rewritten variants. Our method is based on the observation that differences between LLM-rewritten and original code tend to be smaller when the original code is synthetic. We utilize self-supervised contrastive learning to train a code similarity model and evaluate our approach on two synthetic code detection benchmarks. Our results demonstrate a significant improvement over existing SOTA synthetic content detectors, with AUROC scores increasing by 20.5% on the APPS benchmark and 29.1% on the MBPP benchmark.