$\texttt{Droid}$: A Resource Suite for AI-Generated Code Detection
Daniil Orel, Indraneil Paul, Iryna Gurevych, Preslav Nakov

TL;DR
This paper introduces DroidCollection, a large dataset for training and evaluating AI-generated code detectors, and DroidDetect, a suite of detectors that address generalization and robustness issues through adversarial training and advanced learning techniques.
Contribution
The paper provides the largest open dataset for AI code detection and develops robust detectors that improve generalization and resistance to adversarial attacks.
Findings
Existing detectors do not generalize well across different domains and languages.
Detectors can be easily fooled by superficial prompt modifications.
Training on adversarial data and using metric learning improves detection robustness.
Abstract
In this work, we compile \textbf{\texttt{DroidCollection}}, the most extensive open data suite for training and evaluating machine-generated code detectors, comprising over a million code samples, seven programming languages, outputs from 43 coding models, and over three real-world coding domains. Alongside fully AI-generated samples, our collection includes human-AI co-authored code, as well as adversarial samples explicitly crafted to evade detection. Subsequently, we develop \textbf{\texttt{DroidDetect}}, a suite of encoder-only detectors trained using a multi-task objective over . Our experiments show that existing detectors' performance fails to generalise to diverse coding domains and programming languages outside of their narrow training data. Additionally, we demonstrate that while most detectors are easily compromised by humanising the output…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
Taxonomy
TopicsNatural Language Processing Techniques · Text Readability and Simplification · Software Engineering Research
