The power of text similarity in identifying AI-LLM paraphrased documents: The case of BBC news articles and ChatGPT

TL;DR

This paper presents a pattern-based similarity detection method that accurately identifies AI-generated paraphrased news articles, specifically from ChatGPT, with over 96% accuracy, without relying on deep learning techniques.

Contribution

The study introduces a novel pattern similarity algorithm capable of detecting ChatGPT paraphrased news articles, outperforming existing methods in accuracy and specificity.

Findings

Achieved over 96% accuracy in detecting ChatGPT paraphrased articles.

Demonstrated effectiveness of pattern-based detection without deep learning.

Created a benchmark dataset with BBC articles and ChatGPT paraphrases.

Abstract

Generative AI paraphrased text can be used for copyright infringement and the AI paraphrased content can deprive substantial revenue from original content creators. Despite this recent surge of malicious use of generative AI, there are few academic publications that research this threat. In this article, we demonstrate the ability of pattern-based similarity detection for AI paraphrased news recognition. We propose an algorithmic scheme, which is not limited to detect whether an article is an AI paraphrase, but, more importantly, to identify that the source of infringement is the ChatGPT. The proposed method is tested with a benchmark dataset specifically created for this task that incorporates real articles from BBC, incorporating a total of 2,224 articles across five different news categories, as well as 2,224 paraphrased articles created with ChatGPT. Results show that our pattern similarity-based method, that makes no use of deep learning, can detect ChatGPT assisted paraphrased articles at percentages 96.23% for accuracy, 96.25% for precision, 96.21% for sensitivity, 96.25% for specificity and 96.23% for F1 score.