Identifying Machine-Paraphrased Plagiarism

TL;DR

This study evaluates machine learning and neural language models, especially Longformer, for detecting machine-paraphrased plagiarism in academic texts, outperforming traditional text-matching tools.

Contribution

It introduces a comprehensive evaluation of pre-trained embedding models and neural language models for paraphrase detection, providing open data and tools for future research.

Findings

Longformer achieved an average F1 score of 81.0%.

Neural models outperform traditional text-matching systems.

Open datasets and tools are publicly available.

Abstract

Employing paraphrasing tools to conceal plagiarized text is a severe threat to academic integrity. To enable the detection of machine-paraphrased text, we evaluate the effectiveness of five pre-trained word embedding models combined with machine-learning classifiers and eight state-of-the-art neural language models. We analyzed preprints of research papers, graduation theses, and Wikipedia articles, which we paraphrased using different configurations of the tools SpinBot and SpinnerChief. The best-performing technique, Longformer, achieved an average F1 score of 81.0% (F1=99.7% for SpinBot and F1=71.6% for SpinnerChief cases), while human evaluators achieved F1=78.4% for SpinBot and F1=65.6% for SpinnerChief cases. We show that the automated classification alleviates shortcomings of widely-used text-matching systems, such as Turnitin and PlagScan. To facilitate future research, all data, code, and two web applications showcasing our contributions are openly available at https://github.com/jpwahle/iconf22-paraphrase.