Almost AI, Almost Human: The Challenge of Detecting AI-Polished Writing

TL;DR

This paper examines the difficulty of detecting AI-polished human text, revealing that current detectors often misclassify minimally refined content and struggle to assess AI involvement levels, indicating a need for improved detection methods.

Contribution

The study introduces the APT-Eval dataset and systematically evaluates twelve state-of-the-art AI-text detectors on AI-polished content, exposing their limitations.

Findings

Detectors often falsely flag minimally polished human text as AI-generated

Current detectors cannot reliably differentiate degrees of AI involvement

Biases exist against older and smaller AI models in detection accuracy

Abstract

The growing use of large language models (LLMs) for text generation has led to widespread concerns about AI-generated content detection. However, an overlooked challenge is AI-polished text, where human-written content undergoes subtle refinements using AI tools. This raises a critical question: should minimally polished text be classified as AI-generated? Such classification can lead to false plagiarism accusations and misleading claims about AI prevalence in online content. In this study, we systematically evaluate twelve state-of-the-art AI-text detectors using our AI-Polished-Text Evaluation (APT-Eval) dataset, which contains 14.7K samples refined at varying AI-involvement levels. Our findings reveal that detectors frequently flag even minimally polished text as AI-generated, struggle to differentiate between degrees of AI involvement, and exhibit biases against older and smaller models. These limitations highlight the urgent need for more nuanced detection methodologies.