Can You Detect the Difference?

TL;DR

This paper compares diffusion-based and autoregressive language models, revealing that diffusion models closely mimic human text and pose challenges for existing detection methods, emphasizing the need for diffusion-aware detection techniques.

Contribution

It provides the first systematic comparison of diffusion-generated and AR-generated texts, highlighting limitations of current detection metrics and proposing new directions for diffusion-aware detection.

Findings

LLaDA mimics human text closely, causing high false negatives.

LLaMA has lower perplexity but less lexical fidelity.

Single metrics are insufficient for detection.

Abstract

The rapid advancement of large language models (LLMs) has raised concerns about reliably detecting AI-generated text. Stylometric metrics work well on autoregressive (AR) outputs, but their effectiveness on diffusion-based models is unknown. We present the first systematic comparison of diffusion-generated text (LLaDA) and AR-generated text (LLaMA) using 2 000 samples. Perplexity, burstiness, lexical diversity, readability, and BLEU/ROUGE scores show that LLaDA closely mimics human text in perplexity and burstiness, yielding high false-negative rates for AR-oriented detectors. LLaMA shows much lower perplexity but reduced lexical fidelity. Relying on any single metric fails to separate diffusion outputs from human writing. We highlight the need for diffusion-aware detectors and outline directions such as hybrid models, diffusion-specific stylometric signatures, and robust watermarking.