DetectGPT-SC: Improving Detection of Text Generated by Large Language Models through Self-Consistency with Masked Predictions

TL;DR

DetectGPT-SC introduces a novel method leveraging large language models' self-consistency in masked text predictions to improve detection of AI-generated texts, outperforming existing detectors across various tasks.

Contribution

The paper proposes a new detection approach based on self-consistency with masked predictions, exploiting LLMs' reasoning ability to distinguish AI-generated texts from human-written ones.

Findings

DetectGPT-SC outperforms current state-of-the-art detectors.

Self-consistency with masked predictions effectively identifies AI-generated texts.

The method works across different mask schemes and prompts.

Abstract

General large language models (LLMs) such as ChatGPT have shown remarkable success, but it has also raised concerns among people about the misuse of AI-generated texts. Therefore, an important question is how to detect whether the texts are generated by ChatGPT or by humans. Existing detectors are built on the assumption that there is a distribution gap between human-generated and AI-generated texts. These gaps are typically identified using statistical information or classifiers. In contrast to prior research methods, we find that large language models such as ChatGPT exhibit strong self-consistency in text generation and continuation. Self-consistency capitalizes on the intuition that AI-generated texts can still be reasoned with by large language models using the same logical reasoning when portions of the texts are masked, which differs from human-generated texts. Using this observation, we subsequently proposed a new method for AI-generated texts detection based on self-consistency with masked predictions to determine whether a text is generated by LLMs. This method, which we call DetectGPT-SC. We conducted a series of experiments to evaluate the performance of DetectGPT-SC. In these experiments, we employed various mask scheme, zero-shot, and simple prompt for completing masked texts and self-consistency predictions. The results indicate that DetectGPT-SC outperforms the current state-of-the-art across different tasks.