Luminol-AIDetect: Fast Zero-shot Machine-Generated Text Detection based on Perplexity under Text Shuffling

TL;DR

Luminol-AIDetect is a zero-shot, model-agnostic method that detects machine-generated text by analyzing structural fragility through perplexity shifts caused by text shuffling.

Contribution

It introduces a novel, zero-shot approach leveraging perplexity dispersion under text shuffling to distinguish machine-generated from human text without model-specific training.

Findings

Achieves up to 17x lower false positive rate compared to prior methods.

Demonstrates effectiveness across 8 content domains, 11 attack types, and 18 languages.

Operates efficiently without requiring training on specific generation models.

Abstract

Machine-generated text (MGT) detection requires identifying structurally invariant signals across generation models, rather than relying on model-specific fingerprints. In this respect, we hypothesize that while large language models excel at local semantic consistency, their autoregressive nature results in a specific kind of structural fragility compared to human writing. We propose Luminol-AIDetect, a novel, zero-shot statistical approach that exposes this fragility through coherence disruption. By applying a simple randomized text-shuffling procedure, we demonstrate that the resulting shift in perplexity serves as a principled, model-agnostic discriminant, as MGT displays a characteristic dispersion in perplexity-under-shuffling that differs markedly from the more stable structural variability of human-written text. Luminol-AIDetect leverages this distinction to inform its decision process, where a handful of perplexity-based scalar features are extracted from an input text and its shuffled version, then detection is performed via density estimation and ensemble-based prediction. Evaluated across 8 content domains, 11 adversarial attack types, and 18 languages, Luminol-AIDetect demonstrates state-of-the-art performance, with gains up to 17x lower FPR while being cheaper than prior methods.