Deep Learning Detection Method for Large Language Models-Generated Scientific Content

TL;DR

This paper introduces AI-Catcher, a deep learning-based method combining MLP and CNN to detect ChatGPT-generated scientific content, supported by a new dataset and showing significant accuracy improvements.

Contribution

The paper presents a novel multimodal detection approach, AI-Catcher, and a new dataset for distinguishing AI-generated scientific texts from human-written ones.

Findings

AI-Catcher outperforms existing methods in accuracy.

The dataset contains 3000 labeled scientific texts across ten domains.

AI-Catcher improves detection accuracy by an average of 37.4%.

Abstract

Large Language Models (LLMs), such as GPT-3 and BERT, reshape how textual content is written and communicated. These models have the potential to generate scientific content that is indistinguishable from that written by humans. Hence, LLMs carry severe consequences for the scientific community, which relies on the integrity and reliability of publications. This research paper presents a novel ChatGPT-generated scientific text detection method, AI-Catcher. AI-Catcher integrates two deep learning models, multilayer perceptron (MLP) and convolutional neural networks (CNN). The MLP learns the feature representations of the linguistic and statistical features. The CNN extracts high-level representations of the sequential patterns from the textual content. AI-Catcher is a multimodal model that fuses hidden patterns derived from MLP and CNN. In addition, a new ChatGPT-Generated scientific text dataset is collected to enhance AI-generated text detection tools, AIGTxt. AIGTxt contains 3000 records collected from published academic articles across ten domains and divided into three classes: Human-written, ChatGPT-generated, and Mixed text. Several experiments are conducted to evaluate the performance of AI-Catcher. The comparative results demonstrate the capability of AI-Catcher to distinguish between human-written and ChatGPT-generated scientific text more accurately than alternative methods. On average, AI-Catcher improved accuracy by 37.4%.