Transforming Chatbot Text: A Sequence-to-Sequence Approach

TL;DR

This paper presents a sequence-to-sequence method to transform GPT-generated text into more human-like text, reducing detection accuracy and enhancing understanding of AI text manipulation.

Contribution

It introduces a novel Seq2Seq transformation technique that makes GPT-generated text more human-like, impacting both detection and generation of AI text.

Findings

Transformed GPT text is harder to detect with classifiers trained on original data.

Retraining classifiers on transformed data improves detection accuracy.

Seq2Seq models effectively modify linguistic and semantic features of AI-generated text.

Abstract

Due to advances in Large Language Models (LLMs) such as ChatGPT, the boundary between human-written text and AI-generated text has become blurred. Nevertheless, recent work has demonstrated that it is possible to reliably detect GPT-generated text. In this paper, we adopt a novel strategy to adversarially transform GPT-generated text using sequence-to-sequence (Seq2Seq) models, with the goal of making the text more human-like. We experiment with the Seq2Seq models T5-small and BART which serve to modify GPT-generated sentences to include linguistic, structural, and semantic components that may be more typical of human-authored text. Experiments show that classification models trained to distinguish GPT-generated text are significantly less accurate when tested on text that has been modified by these Seq2Seq models. However, after retraining classification models on data generated by our Seq2Seq technique, the models are able to distinguish the transformed GPT-generated text from human-generated text with high accuracy. This work adds to the accumulating knowledge of text transformation as a tool for both attack -- in the sense of defeating classification models -- and defense -- in the sense of improved classifiers -- thereby advancing our understanding of AI-generated text.