Adversarial Attacks on AI-Generated Text Detection Models: A Token Probability-Based Approach Using Embeddings

TL;DR

This paper introduces a novel adversarial attack method on AI-generated text detectors using embedding-based data perturbation, significantly reducing detection accuracy across multiple datasets.

Contribution

It proposes a new token probability-based adversarial attack leveraging embeddings and synonyms to evade AI-generated text detection models.

Findings

Detection scores decreased significantly on XSum and SQuAD datasets.

The method achieves state-of-the-art reduction in detection accuracy.

Embedding techniques effectively mislead detection models.

Abstract

In recent years, text generation tools utilizing Artificial Intelligence (AI) have occasionally been misused across various domains, such as generating student reports or creative writings. This issue prompts plagiarism detection services to enhance their capabilities in identifying AI-generated content. Adversarial attacks are often used to test the robustness of AI-text generated detectors. This work proposes a novel textual adversarial attack on the detection models such as Fast-DetectGPT. The method employs embedding models for data perturbation, aiming at reconstructing the AI generated texts to reduce the likelihood of detection of the true origin of the texts. Specifically, we employ different embedding techniques, including the Tsetlin Machine (TM), an interpretable approach in machine learning for this purpose. By combining synonyms and embedding similarity vectors, we demonstrates the state-of-the-art reduction in detection scores against Fast-DetectGPT. Particularly, in the XSum dataset, the detection score decreased from 0.4431 to 0.2744 AUROC, and in the SQuAD dataset, it dropped from 0.5068 to 0.3532 AUROC.