Sure! Here's a short and concise title for your paper: "Contamination in Generated Text Detection Benchmarks"

TL;DR

This paper highlights the importance of high-quality datasets for AI-generated text detection, demonstrating that data cleansing reduces detector vulnerability to spoofing, and provides a reprocessed, improved benchmark dataset.

Contribution

It introduces a data cleansing process for existing benchmarks, improving their robustness against spoofing attacks and making the dataset publicly available.

Findings

Data cleansing reduces detector vulnerability to spoofing.

Reprocessed dataset is more challenging for detectors.

Publicly available improved benchmark dataset.

Abstract

Large language models are increasingly used for many applications. To prevent illicit use, it is desirable to be able to detect AI-generated text. Training and evaluation of such detectors critically depend on suitable benchmark datasets. Several groups took on the tedious work of collecting, curating, and publishing large and diverse datasets for this task. However, it remains an open challenge to ensure high quality in all relevant aspects of such a dataset. For example, the DetectRL benchmark exhibits relatively simple patterns of AI-generation in 98.5% of the Claude-LLM data. These patterns may include introductory words such as "Sure! Here is the academic article abstract:", or instances where the LLM rejects the prompted task. In this work, we demonstrate that detectors trained on such data use such patterns as shortcuts, which facilitates spoofing attacks on the trained detectors. We consequently reprocessed the DetectRL dataset with several cleansing operations. Experiments show that such data cleansing makes direct attacks more difficult. The reprocessed dataset is publicly available.