Cat and Mouse -- Can Fake Text Generation Outpace Detector Systems?

TL;DR

This paper investigates whether large language models can outpace detection systems for fake text, finding that simpler classifiers can still effectively identify deceptive text despite advancements in model size and complexity.

Contribution

It demonstrates that detection of fake text remains feasible with modest classifiers even as models grow larger and more sophisticated.

Findings

Gemini's ability to generate deceptive text increased with model size.

GPT's deception ability did not significantly improve with size.

Detection methods may remain effective despite larger, more complex models.

Abstract

Large language models can produce convincing "fake text" in domains such as academic writing, product reviews, and political news. Many approaches have been investigated for the detection of artificially generated text. While this may seem to presage an endless "arms race", we note that newer LLMs use ever more parameters, training data, and energy, while relatively simple classifiers demonstrate a good level of detection accuracy with modest resources. To approach the question of whether the models' ability to beat the detectors may therefore reach a plateau, we examine the ability of statistical classifiers to identify "fake text" in the style of classical detective fiction. Over a 0.5 version increase, we found that Gemini showed an increased ability to generate deceptive text, while GPT did not. This suggests that reliable detection of fake text may remain feasible even for ever-larger models, though new model architectures may improve their deceptiveness