Quantitative Analysis of AI-Generated Texts in Academic Research: A Study of AI Presence in Arxiv Submissions using AI Detection Tool

TL;DR

This paper evaluates the effectiveness of AI detection tools, specifically Originality.ai, in identifying AI-generated content in Arxiv research submissions across physics, mathematics, and computer science.

Contribution

It introduces a new dataset of Arxiv articles and assesses the accuracy of Originality.ai in detecting AI-generated texts.

Findings

Originality.ai achieves 98% detection accuracy.

The dataset includes physics, mathematics, and computer science articles.

AI detection tools can reliably identify AI-generated academic content.

Abstract

Many people are interested in ChatGPT since it has become a prominent AIGC model that provides high-quality responses in various contexts, such as software development and maintenance. Misuse of ChatGPT might cause significant issues, particularly in public safety and education, despite its immense potential. The majority of researchers choose to publish their work on Arxiv. The effectiveness and originality of future work depend on the ability to detect AI components in such contributions. To address this need, this study will analyze a method that can see purposely manufactured content that academic organizations use to post on Arxiv. For this study, a dataset was created using physics, mathematics, and computer science articles. Using the newly built dataset, the following step is to put originality.ai through its paces. The statistical analysis shows that Originality.ai is very accurate, with a rate of 98%.