Who Writes What: Unveiling the Impact of Author Roles on AI-generated Text Detection

TL;DR

This study reveals that author sociolinguistic attributes like proficiency and environment significantly influence AI-generated text detection accuracy, emphasizing the need for socially aware detection methods to ensure fairness.

Contribution

It provides the first empirical analysis of how demographic factors impact AI text detector biases, introducing a robust statistical framework for evaluating fairness.

Findings

CEFR proficiency affects detector accuracy

Language environment influences detection bias

Gender and academic field effects vary by detector

Abstract

The rise of Large Language Models (LLMs) necessitates accurate AI-generated text detection. However, current approaches largely overlook the influence of author characteristics. We investigate how sociolinguistic attributes-gender, CEFR proficiency, academic field, and language environment-impact state-of-the-art AI text detectors. Using the ICNALE corpus of human-authored texts and parallel AI-generated texts from diverse LLMs, we conduct a rigorous evaluation employing multi-factor ANOVA and weighted least squares (WLS). Our results reveal significant biases: CEFR proficiency and language environment consistently affected detector accuracy, while gender and academic field showed detector-dependent effects. These findings highlight the crucial need for socially aware AI text detection to avoid unfairly penalizing specific demographic groups. We offer novel empirical evidence, a robust statistical framework, and actionable insights for developing more equitable and reliable detection systems in real-world, out-of-domain contexts. This work paves the way for future research on bias mitigation, inclusive evaluation benchmarks, and socially responsible LLM detectors.