Complete Evasion, Zero Modification: PDF Attacks on AI Text Detection

TL;DR

PDFuzz introduces a novel PDF attack that manipulates character positioning to evade AI text detectors without altering visual appearance, exposing a structural vulnerability in current detection methods.

Contribution

The paper presents PDFuzz, a new attack method exploiting PDF structure to completely evade AI text detectors while preserving visual fidelity.

Findings

Detector accuracy drops from 93.6% to 50.4% under attack

F1 score drops from 0.938 to 0.0, indicating failure of detection

Attack maintains perfect visual fidelity of the original document

Abstract

AI-generated text detectors have become essential tools for maintaining content authenticity, yet their robustness against evasion attacks remains questionable. We present PDFuzz, a novel attack that exploits the discrepancy between visual text layout and extraction order in PDF documents. Our method preserves exact textual content while manipulating character positioning to scramble extraction sequences. We evaluate this approach against the ArguGPT detector using a dataset of human and AI-generated text. Our results demonstrate complete evasion: detector performance drops from (93.6 $\pm$ 1.4) % accuracy and 0.938 $\pm$ 0.014 F1 score to random-level performance ((50.4 $\pm$ 3.2) % accuracy, 0.0 F1 score) while maintaining perfect visual fidelity. Our work reveals a vulnerability in current detection systems that is inherent to PDF document structures and underscores the need for implementing sturdy safeguards against such attacks. We make our code publicly available at https://github.com/ACMCMC/PDFuzz.