Is this Snippet Written by ChatGPT? An Empirical Study with a CodeBERT-Based Classifier

TL;DR

This paper introduces GPTSniffer, a CodeBERT-based classifier, to detect AI-generated code snippets, demonstrating high accuracy and outperforming existing tools, with factors like training data similarity influencing performance.

Contribution

The paper presents GPTSniffer, a novel AI code detection method that improves accuracy over existing tools and analyzes factors affecting classification performance.

Findings

GPTSniffer outperforms GPTZero and OpenAI Text Classifier.

Training data similarity boosts detection accuracy.

Paired snippets in classification context improve results.

Abstract

Since its launch in November 2022, ChatGPT has gained popularity among users, especially programmers who use it as a tool to solve development problems. However, while offering a practical solution to programming problems, ChatGPT should be mainly used as a supporting tool (e.g., in software education) rather than as a replacement for the human being. Thus, detecting automatically generated source code by ChatGPT is necessary, and tools for identifying AI-generated content may need to be adapted to work effectively with source code. This paper presents an empirical study to investigate the feasibility of automated identification of AI-generated code snippets, and the factors that influence this ability. To this end, we propose a novel approach called GPTSniffer, which builds on top of CodeBERT to detect source code written by AI. The results show that GPTSniffer can accurately classify whether code is human-written or AI-generated, and outperforms two baselines, GPTZero and OpenAI Text Classifier. Also, the study shows how similar training data or a classification context with paired snippets helps to boost classification performances.