Detecting Machine-Generated Long-Form Content with Latent-Space Variables

TL;DR

This paper introduces a robust detection method for machine-generated long-form content by analyzing event transitions in latent space, outperforming existing token-based detectors and revealing inherent differences in event generation between humans and LLMs.

Contribution

The paper presents a novel latent-space model that leverages event transitions to distinguish machine from human texts, improving detection accuracy under domain shifts.

Findings

31% improvement over DetectGPT in distinguishing texts

Modern LLMs generate event triggers differently from humans

Latent-space analysis reveals inherent differences in event transitions

Abstract

The increasing capability of large language models (LLMs) to generate fluent long-form texts is presenting new challenges in distinguishing machine-generated outputs from human-written ones, which is crucial for ensuring authenticity and trustworthiness of expressions. Existing zero-shot detectors primarily focus on token-level distributions, which are vulnerable to real-world domain shifts, including different prompting and decoding strategies, and adversarial attacks. We propose a more robust method that incorporates abstract elements, such as event transitions, as key deciding factors to detect machine versus human texts by training a latent-space model on sequences of events or topics derived from human-written texts. In three different domains, machine-generated texts, which are originally inseparable from human texts on the token level, can be better distinguished with our latent-space model, leading to a 31% improvement over strong baselines such as DetectGPT. Our analysis further reveals that, unlike humans, modern LLMs like GPT-4 generate event triggers and their transitions differently, an inherent disparity that helps our method to robustly detect machine-generated texts.