Assaying on the Robustness of Zero-Shot Machine-Generated Text Detectors

TL;DR

This paper investigates the robustness of zero-shot machine-generated text detectors across different topics and LLMs, highlighting their limitations and the impact of topic shifts on detection performance.

Contribution

It provides empirical insights into how topic variability affects zero-shot detectors and explores the robustness of various advanced LLMs in this context.

Findings

Detection performance correlates with topics.

Topic shifts significantly impact detection accuracy.

Robustness varies across different LLMs.

Abstract

To combat the potential misuse of Natural Language Generation (NLG) technology, a variety of algorithms have been developed for the detection of AI-generated texts. Traditionally, this task is treated as a binary classification problem. Although supervised learning has demonstrated promising results, acquiring labeled data for detection purposes poses real-world challenges and the risk of overfitting. In an effort to address these issues, we delve into the realm of zero-shot machine-generated text detection. Existing zero-shot detectors, typically designed for specific tasks or topics, often assume uniform testing scenarios, limiting their practicality. In our research, we explore various advanced Large Language Models (LLMs) and their specialized variants, contributing to this field in several ways. In empirical studies, we uncover a significant correlation between topics and detection performance. Secondly, we delve into the influence of topic shifts on zero-shot detectors. These investigations shed light on the adaptability and robustness of these detection methods across diverse topics. The code is available at \url{https://github.com/yfzhang114/robustness-detection}.