Fighting Fire with Fire: Can ChatGPT Detect AI-generated Text?

TL;DR

This paper evaluates ChatGPT's ability to detect AI-generated text in a zero-shot setting, exploring its effectiveness and potential role in automated detection systems.

Contribution

It investigates ChatGPT's zero-shot performance as a detector for AI-generated text and provides insights into leveraging LLMs for detection tasks.

Findings

ChatGPT performs reasonably well in detecting AI-generated text.

Detection effectiveness varies between AI-generated and human-written text.

The study offers practical insights for integrating LLMs into detection pipelines.

Abstract

Large language models (LLMs) such as ChatGPT are increasingly being used for various use cases, including text content generation at scale. Although detection methods for such AI-generated text exist already, we investigate ChatGPT's performance as a detector on such AI-generated text, inspired by works that use ChatGPT as a data labeler or annotator. We evaluate the zero-shot performance of ChatGPT in the task of human-written vs. AI-generated text detection, and perform experiments on publicly available datasets. We empirically investigate if ChatGPT is symmetrically effective in detecting AI-generated or human-written text. Our findings provide insight on how ChatGPT and similar LLMs may be leveraged in automated detection pipelines by simply focusing on solving a specific aspect of the problem and deriving the rest from that solution. All code and data is available at https://github.com/AmritaBh/ChatGPT-as-Detector.