On the Generalization and Adaptation Ability of Machine-Generated Text Detectors in Academic Writing

TL;DR

This paper investigates the generalization and adaptation capabilities of machine-generated text detectors in academic writing, introducing a large dataset, benchmarking various detectors, and proposing methods for adaptive detection in evolving scenarios.

Contribution

It presents MGT-Acedemic, a large-scale academic writing dataset, benchmarks detector performance across domains, and introduces an adaptive attribution task with multiple techniques.

Findings

Detectors face challenges in attribution tasks across domains.

Adaptive techniques improve detection performance in limited data scenarios.

The study highlights the complexity of generalizing MGT detection in academic contexts.

Abstract

The rising popularity of large language models (LLMs) has raised concerns about machine-generated text (MGT), particularly in academic settings, where issues like plagiarism and misinformation are prevalent. As a result, developing a highly generalizable and adaptable MGT detection system has become an urgent priority. Given that LLMs are most commonly misused in academic writing, this work investigates the generalization and adaptation capabilities of MGT detectors in three key aspects specific to academic writing: First, we construct MGT-Acedemic, a large-scale dataset comprising over 336M tokens and 749K samples. MGT-Acedemic focuses on academic writing, featuring human-written texts (HWTs) and MGTs across STEM, Humanities, and Social Sciences, paired with an extensible code framework for efficient benchmarking. Second, we benchmark the performance of various detectors for binary classification and attribution tasks in both in-domain and cross-domain settings. This benchmark reveals the often-overlooked challenges of attribution tasks. Third, we introduce a novel attribution task where models have to adapt to new classes over time without (or with very limited) access to prior training data in both few-shot and many-shot scenarios. We implement eight different adapting techniques to improve the performance and highlight the inherent complexity of the task. Our findings provide insights into the generalization and adaptation ability of MGT detectors across diverse scenarios and lay the foundation for building robust, adaptive detection systems. The code framework is available at https://github.com/Y-L-LIU/MGTBench-2.0.