LLM-Generated or Human-Written? Comparing Review and Non-Review Papers on ArXiv

TL;DR

This study quantifies the prevalence of LLM-generated content in arXiv papers, revealing a significant increase especially in review papers, and discusses the implications of arXiv's policy to ban unpublished review submissions.

Contribution

It provides the first quantitative analysis of LLM-generated content in arXiv papers, comparing review and non-review categories with two detection methods.

Findings

LLM-generated content has increased substantially in recent years.

Higher prevalence of LLM-generated content in review papers.

Non-review LLM-generated papers are nearly six times more numerous.

Abstract

ArXiv recently prohibited the upload of unpublished review papers to its servers in the Computer Science domain, citing a high prevalence of LLM-generated content in these categories. However, this decision was not accompanied by quantitative evidence. In this work, we investigate this claim by measuring the proportion of LLM-generated content in review vs. non-review research papers in recent years. Using two high-quality detection methods, we find a substantial increase in LLM-generated content across both review and non-review papers, with a higher prevalence in review papers. However, when considering the number of LLM-generated papers published in each category, the estimates of non-review LLM-generated papers are almost six times higher. Furthermore, we find that this policy will affect papers in certain domains far more than others, with the CS subdiscipline Computers & Society potentially facing cuts of 50%. Our analysis provides an evidence-based framework for evaluating such policy decisions, and we release our code to facilitate future investigations at: https://github.com/yanaiela/llm-review-arxiv.