PlagBench: Exploring the Duality of Large Language Models in Plagiarism Generation and Detection

TL;DR

This paper introduces PlagBench, a large dataset of synthetic plagiarism examples generated by LLMs, to evaluate LLMs' abilities in both creating and detecting various types of plagiarism, revealing significant detection improvements with GPT-4.

Contribution

The paper provides a new dataset, PlagBench, for studying LLMs in plagiarism generation and detection, and offers comprehensive evaluation of LLMs' capabilities in these tasks.

Findings

GPT-3.5 Turbo generates high-quality paraphrases and summaries.

GPT-4 outperforms other LLMs and detection tools by 20%.

LLMs show evolving abilities in both content creation and plagiarism detection.

Abstract

Recent studies have raised concerns about the potential threats large language models (LLMs) pose to academic integrity and copyright protection. Yet, their investigation is predominantly focused on literal copies of original texts. Also, how LLMs can facilitate the detection of LLM-generated plagiarism remains largely unexplored. To address these gaps, we introduce \textbf{{\sf PlagBench}}, a dataset of 46.5K synthetic text pairs that represent three major types of plagiarism: verbatim copying, paraphrasing, and summarization. These samples are generated by three advanced LLMs. We rigorously validate the quality of PlagBench through a combination of fine-grained automatic evaluation and human annotation. We then utilize this dataset for two purposes: (1) to examine LLMs' ability to transform original content into accurate paraphrases and summaries, and (2) to evaluate the plagiarism detection performance of five modern LLMs alongside three specialized plagiarism checkers. Our results show that GPT-3.5 Turbo can produce high-quality paraphrases and summaries without significantly increasing text complexity compared to GPT-4 Turbo. However, in terms of detection, GPT-4 outperforms other LLMs and commercial detection tools by 20%, highlights the evolving capabilities of LLMs not only in content generation but also in plagiarism detection. Data and source code are available at https://github.com/Brit7777/plagbench.