Different Time, Different Language: Revisiting the Bias Against Non-Native Speakers in GPT Detectors

TL;DR

This paper investigates whether GPT detectors are biased against non-native Czech speakers and finds that modern detectors do not rely on perplexity and are not systematically biased.

Contribution

The study revisits the bias against non-native speakers in GPT detectors, showing that in Czech, detectors operate effectively without relying on perplexity and lack systematic bias.

Findings

Perplexity of non-native Czech texts is not lower than native texts.

Detectors from three families show no systematic bias against non-native speakers.

Contemporary detectors do not rely on perplexity for detection.

Abstract

LLM-based assistants have been widely popularised after the release of ChatGPT. Concerns have been raised about their misuse in academia, given the difficulty of distinguishing between human-written and generated text. To combat this, automated techniques have been developed and shown to be effective, to some extent. However, prior work suggests that these methods often falsely flag essays from non-native speakers as generated, due to their low perplexity extracted from an LLM, which is supposedly a key feature of the detectors. We revisit these statements two years later, specifically in the Czech language setting. We show that the perplexity of texts from non-native speakers of Czech is not lower than that of native speakers. We further examine detectors from three separate families and find no systematic bias against non-native speakers. Finally, we demonstrate that contemporary detectors operate effectively without relying on perplexity.