GradEscape: A Gradient-Based Evader Against AI-Generated Text Detectors

TL;DR

GradEscape is a novel gradient-based method that effectively evades AI-generated text detectors by overcoming discrete text challenges, adapting across models, and demonstrating superior performance in real-world scenarios.

Contribution

We introduce GradEscape, the first gradient-based evader that addresses the discrete nature of text and adapts to various detectors, enhancing evasion success.

Findings

GradEscape outperforms existing evaders on multiple datasets and models.

It successfully evades commercial AI-generated text detectors.

The primary vulnerability is linked to style disparities in training data.

Abstract

In this paper, we introduce GradEscape, the first gradient-based evader designed to attack AI-generated text (AIGT) detectors. GradEscape overcomes the undifferentiable computation problem, caused by the discrete nature of text, by introducing a novel approach to construct weighted embeddings for the detector input. It then updates the evader model parameters using feedback from victim detectors, achieving high attack success with minimal text modification. To address the issue of tokenizer mismatch between the evader and the detector, we introduce a warm-started evader method, enabling GradEscape to adapt to detectors across any language model architecture. Moreover, we employ novel tokenizer inference and model extraction techniques, facilitating effective evasion even in query-only access. We evaluate GradEscape on four datasets and three widely-used language models, benchmarking it against four state-of-the-art AIGT evaders. Experimental results demonstrate that GradEscape outperforms existing evaders in various scenarios, including with an 11B paraphrase model, while utilizing only 139M parameters. We have successfully applied GradEscape to two real-world commercial AIGT detectors. Our analysis reveals that the primary vulnerability stems from disparity in text expression styles within the training data. We also propose a potential defense strategy to mitigate the threat of AIGT evaders. We open-source our GradEscape for developing more robust AIGT detectors.