$\texttt{Droid}$: A Resource Suite for AI-Generated Code Detection

TL;DR

This paper introduces DroidCollection, a large dataset for training and evaluating AI-generated code detectors, and DroidDetect, a suite of detectors that address generalization and robustness issues through adversarial training and advanced learning techniques.

Contribution

The paper provides the largest open dataset for AI code detection and develops robust detectors that improve generalization and resistance to adversarial attacks.

Findings

Existing detectors do not generalize well across different domains and languages.

Detectors can be easily fooled by superficial prompt modifications.

Training on adversarial data and using metric learning improves detection robustness.

Abstract

In this work, we compile $\textbf{$\texttt{DroidCollection}$}$, the most extensive open data suite for training and evaluating machine-generated code detectors, comprising over a million code samples, seven programming languages, outputs from 43 coding models, and over three real-world coding domains. Alongside fully AI-generated samples, our collection includes human-AI co-authored code, as well as adversarial samples explicitly crafted to evade detection. Subsequently, we develop $\textbf{$\texttt{DroidDetect}$}$, a suite of encoder-only detectors trained using a multi-task objective over $\texttt{DroidCollection}$. Our experiments show that existing detectors' performance fails to generalise to diverse coding domains and programming languages outside of their narrow training data. Additionally, we demonstrate that while most detectors are easily compromised by humanising the output distributions using superficial prompting and alignment approaches, this problem can be easily amended by training on a small amount of adversarial data. Finally, we demonstrate the effectiveness of metric learning and uncertainty-based resampling as means to enhance detector training on possibly noisy distributions.