Smaller Language Models are Better Black-box Machine-Generated Text Detectors

TL;DR

This paper shows that smaller, partially-trained language models are more effective at detecting machine-generated text across various models, regardless of training data similarity.

Contribution

It introduces a black-box detection method that leverages smaller models, demonstrating their superior ability to identify generated text from larger models.

Findings

Smaller models outperform larger ones in detection accuracy.

Detection success is not heavily dependent on shared training data.

Smaller models achieve higher AUC scores across different generator models.

Abstract

With the advent of fluent generative language models that can produce convincing utterances very similar to those written by humans, distinguishing whether a piece of text is machine-generated or human-written becomes more challenging and more important, as such models could be used to spread misinformation, fake news, fake reviews and to mimic certain authors and figures. To this end, there have been a slew of methods proposed to detect machine-generated text. Most of these methods need access to the logits of the target model or need the ability to sample from the target. One such black-box detection method relies on the observation that generated text is locally optimal under the likelihood function of the generator, while human-written text is not. We find that overall, smaller and partially-trained models are better universal text detectors: they can more precisely detect text generated from both small and larger models. Interestingly, we find that whether the detector and generator were trained on the same data is not critically important to the detection success. For instance the OPT-125M model has an AUC of 0.81 in detecting ChatGPT generations, whereas a larger model from the GPT family, GPTJ-6B, has AUC of 0.45.